Sunday, May 20, 2018

Linux, trading, security (or Haas, Ubuntu and commitment)

Ubuntu plus Haasbot: Step two

Updated (20) 2018
Busy people: skip to "Making modifications"

Foreword

In the previous article, which proven itself worthy, we described how to install Haas trade server on remote server and start it. As many people rightfully pointed, no security configuration were made whatsoever; in this article I will address the issue.
Please note that while you can do ANY, EVERY or NONE of the suggested modifications, it is VERY important to finish modification you started. Not walking through a single modification to the end might render your server unusable.

Introduction and disclaimer

Security is a relative term in its nature; and what is "secure" for home use, will not do for commercial company. In this article I am going to address MINIMAL package, which complies with the following:
  • Easy to install
  • Requires zero or almost zero maintenance
  • Won't hurt usability
  • Require no commercial or out-of-the-repository software
My point here is that security must NOT interfere with the main functionality of the system you are securing; and having some security is better than having none.

If you want to dive in yourself and install industry-grade security for your server feel free to read and use this awesome article from Linode (their guides are pretty good, but I don't like their hosting service), plus throw in an IP filter and trusted VPN (keep in mind VPN and VPS are not the same thing).

Also note that I have no information for the Haas security testing protocol. Once I will know it - I will update the document, either disabling remote IP usage, or highlighting the security used.

Modifications in question

In this section I will give some explanation and reasoning for them being done. If you are eager to just into action skip to the next section.

  1. Update and upgrade.
    Importance: of utter importance
    Time: 2-20 minutes
    The "update" is the easiest and very safe step to do by far. It should be done whenever update is ready (or automatically even - more of this in later posts); when the "Upgrade" part is a bit tricky. Most often (I'd say 9999 out of 10000 times) this will do good. On the other side, sometimes upgrade can change some internal logic of third-party frameworks\application  (for example - Mono), and mess with your software (HaasTradeServer). Another argument against going all-in on upgrade might be the fact that Linux is known to require very little maintenance; also, Ubuntu will not surprise you like other (*cough-cough-Windows*) OS can:
    Bleh :(
    Verdict: "update" and "security upgrade" must be done as soon as possible, for "upgrade" pick up a good moment every month or so and do this too
  2. Changing ROOT password.
    Importance: good to have
    Time: 2 minutes
    While you might argue that the secure way is to change NOT the password, but actual user allowed to log in through SSH remote user (and you are very true in this saying), I still will point out the compexity of the operation. If you like to change the user, please refer to the article from the "disclaimer" section. The whole point of changing password is just to make sure nobody can get automatically-generated password from your hoster (most probably you received a email with password, username and IP after initial Ubuntu setup).  So by changing password you are making sure this email is no longer relevant; however PLEASE remember the new password or write it down in secure place.
  3. Changing default SSH port
    Importance: a bonus one; can be skipped
    Time: 10 minutes
    This move is a bit controversial, and the reasons for the controversy are well-described in this article. However judging by my IT experience it MIGHT prevent future (highly improbable, but still) hack - if your server got into wannabe-hacker list with "open SSH port" remark, there is a small chance that some unexpected future exploit will come out that will allow breaking into SSH, and your server will be exploited.
  4. Changing default HTS port
    Importance: a bonus one; can be skipped
    Time: 10 minutes
    This move basically mirrors previous one, but for HTS itself. Right now HTS is using rare custom port, and hackers community completely unaware of the product. However I give 6~10 months untill unformation will spread, and HTS server hack will become a thing. So to be ready for that the least we can do is to move from well-known port.
  5. Setting up firewall
    Importance: utterly important
    Time: 2 minutes
    Every Linux comes with a pre-installed firewall (called iptables), but most usually it is deactivated by default. While Linux is better designed in terms of network performance than Windows, it is still important to configure firewall. I will use the UFW - Uncomplicated FireWall product - which is, in fact, NOT a firewall, but just a "command interface" for default Linux firewall.
  6. Fail2ban security program
    Importance: good to have
    Time: 5 minutes
    This program will ban IP of a user trying incorrect password too many times. Really good program, with a lot of options, which are desribed in good details here. However we will go with easy default set up.

Making modifications

Step 0: insurance.
To protect you from yourself please take these precautions first:
- Back up your HTS config\bots\keys\etc as described in this article
- Make an actual note (text file or a paper notepad) and write down there the following information:
Server IP address
Server SSH port (default:22, write new one if you decide to change it)
Root password (write both old one and new one if you decide to change it)
Server HTS ports (defaults are 8090 and 8092; write down new ones if you decide to change it)
Please, DO do it.
  1. Update and upgrade
    If you log in to your server and see message about packages that can be "updated" - it's a no brainer.

    Just fire up the command and see what's in there for you:
    sudo apt update
    Now we see that a lot of packages need "upgrading"; let's do just that:
    sudo apt upgrade
    This may or may not take some time:

    Please note that while NOT required, it surely is a good idea to also restart HaasBot afterwards:
    sudo killall monothen
    cd /opt/haasand finally
    sudo ./Haasbot.sh

  2. Changing Root password
    To do that simply execute:
    passwd

    Enter new password and confirm. NOTICE: that input is NOT reflected in terminal for security reasons. No asterisks, no actual letters. No feedback at all, which might be confusing.

    You can paste password, too, for example your password-manager-generated one.
    I HIGHLY recommend using VERY-VERY secure password, as it's the first line of defense. If you can't think of one - feel free to use special service like LastPass or any other resource
    Write down new pass to your "insurance" note.
  3. Changing default SSH port - PROCEED WITH EXTREME CAUTION
    This step will make your server invisible for the 99% of wannabe-hackers and "Mr. Robot" fans (this is a decent and valid TV show in terms of actual hacking representation, by the way)
    BE WARNED: if you mess up any of the steps most probably server will be locked out forever and will need a complete re-install. So, being careful, type the command to edit SSH daemon config:
    nano /etc/ssh/sshd_config
    to open the config file

    This will open nano editor, which should be used to locate #Port 22 line and remove # sign as well as changing number to something else, from range 1024 - 65535 while trying to avoid "privileged" or "cool" number like 2222 or 3333 or 4242 or something like that. Script kidos are using pre-configured port scanners, and scan port 22 for vulnerabilities; advanced kids also scan "cool" ports. So picking 8923 is a better idea than 1337 Other than that actual number does not matter.
    WARNING: Port can NOT coincide with other taken port. So far HTS is using 8090 and 8092, so keep away from those.
    Initial version:

    What we need to have (use your port number instead of 4022)

    Then press Ctrl+X followed by Y and then Enter. After that it's time to restart the SSH daemon and reconnect by running the command
    service sshd restart
    which should not give any output; after that we need to exit the connection and re-configure Termius by running
    exit

    If everything was done correctly, server becomes unreachable from Termius for now

    To fix that we need to "edit" host settings and change "port" to the value we've chosen - IMPORTANT - remember to DELETE default value "22" in the field first.

    NONE-TERMIUS USE: if you use other tools (putty, console, issh or else) you will need to change port, too. If there is no additional field for port just use IP:PORT syntax. Example for my server will be 136.243.66.13:4022 instead of just 136.243.66.13

    My choice of 4022 port is not fitting beautifully, but works just fine. Save, reconnect and you might need to re-accept server fingerprint

    Write down new port to your "insurance" note.
  4. Changing HTS default port
    Back to basics! Changing HTS config once more. Easy-peasy, just run:
    nano /root/HTS/Settings/MainSettings.XML
    And now go and change <HostingPort> and <HubPort> values. Same drill as with SSH port - pick any ports from range 1024 - 65535 while trying to avoid "privileged" or "cool" number like 2222 or 3333 or 4242 or something like that.
    WARNING: no port can coincide with any other taken port. If you changed SSH port - you can't use same value for HTS.
    Once you pick new numbers (in my example I've picked 48090 and 48092) and wrote them into editor do old trick: Press CTRL+X followed by Y followed by ENTER

    Then go and restart HTS
    sudo killall mono
    then
    cd /opt/haasand finally
    sudo ./Haasbot.sh

    Now write new ports down into your "insurance" note
  5. Firewall control - UFW - PROCEED WITH EXTREME CAUTION
    For the obsolete versions of Ubuntu (16, 17) you might need to use sudo apt install ufw command.
    This modification requires your "insurance" note information. Inability to use correct ports in this step will lock your server out forever and require complete reinstall.
    Our Firewall will allow only HTS and SSH connections, and disable every single other one.
    The number in the following command is YOUR SSH PORT from the "insurance" note. If your SSH port were not modified then use port 22; otherwise use the number of new SSH port from your "insurance" note.
    I have changed mine to 4022, so the command for me is:
    sudo ufw limit 4022
    using "limit" here will not allow to generate too many connections. As I use only one connection at a time, it might help with security

    Then we need to allow our HTS ports to work (again, consult with your "insurance" note for the correct ports) - I have changed mine from 8090 to 48090, so the commands are
    sudo ufw allow 48090
    and
    sudo ufw allow 48092
    now only two last steps left 
    We need to disable every other possible connectivity now by default with:
    sudo ufw default deny incoming
    And finally enable firewall
    sudo ufw enable
    and then press Y and Enter

    To make it start.
    If your server is still accessible and HTS works - you are doing everything just as required =)
  6. Fail2ban
    The last for today - installing Fail2ban to ban those who fail to log in many times in order to prevent password bruteforce attack. Simply type
    sudo apt install fail2ban
    and then press Y

    After really small and fast download program will automatically install and start

    You can check it is working by running:
    fail2ban-client status
    Answer about number of jails indicate succesfull run
Which covers today's topic! 
Please be safe, ask questions and comment on what's not clear in this post!

Advanced Haas tricks - INCOMPLETE

Hey there, it's a work in progress.
Right now I need this article for the sole purpose of HTS old configuration backup. Will be expanded with time.
Updated (20) 2018

Backing up HTS important files

Backing up your Haas server configuration to home computer. Luckily, Haas guys have made it is very easy. If you perform the backup you will retain your bots, configs, exchange keys, license and trade history. The only thing is needed for this is Cyberduck program.
  1. Open Cyberduck and connect to your server using Bookmark

    or any other way you like)
  2. Using "folder" dropdown go to root folder

    And then double-click it
  3. Now from the root directory right-click the HTS folder and choose where to download it to your computer (might take some time)
  4. IMPORTANT: make sure it was downloaded successfully (the message is "Download complete", the bulb is green and size is equal, like "47.8 MB of 47.8 MB")

    And that's all folks!

Thursday, May 10, 2018

Haas Trade Server Ubuntu 18.04 installation comics (Hetzner edition)

Haasbot Ubuntu installation

Updated (21) 2018 - added VisualBasic for email bot

Foreword:

Do not be frightened by the installation. The process is pretty straightforward, easy, and requires NO programming knowledge. Your sole requirement is to pay attention to details and do not hurry.
This document is made for guys who are not very technically savvy. If you are a power user\sysadmin feel free to skim through and perform modifications of your choice.

Introduction.

OS in this tutorial is Ubuntu 18.04 "Minimal" (however almost all of the steps will work on Ubuntu 16.04.4, 17.10 and also should work for other Debian-based Linux distribution).
I will be using free, easy to use software only.
  • I have chosen Ubuntu because it's pretty user-friendly, wide-spread and stable for any task HaasTradeServer may throw. 
  • Version 18 was picked because it is latest LongTermSupport distribution (meaning "reliable" and "no hassle").
  • "Minimal" config with no GUI is chosen to make server less complicated and more robust and secure in future use.
  • Hetzner hosting is used because of the cheap AND fast servers it provides at "Server Auction" Hetzner Servers

For connecting to the remote server we will be using SSH with terminal (console) connection. SSH stands for "secure shell" and it is a cryptographic (ha! Crypto here, too) protocol that is virtually impossible to hack. So your communication with the remote server is secure.
Every time I refer to SSH I will be meaning "this window from Matrix with a lot of letters" - a "terminal console" that will allow you to connect to remote server using SSH protocol.
Your typical Termius SSH console
Using Termius - http://termius.com - for SSH is easier than usual terminal or Putty; but of course, it is up for you to decide which program to utilize.

Terminology

Some terms, such as "login", "password" and "account" are quite confusing, because you can have a lot of them. So I will be calling them, respectively:
  • Hostinglogin (the one you use on Hetzner\VULTR\Linode\Contabo\DigitalOcean\Strato website)
  • SSHlogin (the one you will be using to connect using SSH to the remote server)
  • SSHpassword
  • Haaslogin (the one you will be using to login to http://haasonline.com site)
  • HTSlogin (the one you will be using to enter your HTS application)
  • Exchangelogin - the one you will be using to log in to your exchange.
  • HTSserver - is a computer where HTS is to be installed. Most often it is the IP address of the computer.
  • Laptop - just to make it easier I will refer to your daily computer as a "laptop". It might be desktop or nettop of course.
  • Linux and Ubuntu. While every Ubuntu is a Linux - not every Linux is Ubuntu. To keep confusion to minimum in this document we can use words as a synonims. So, only here and today only Ubuntu=Linux

WARNING Linux commands WARNING

It is VERY important to enter Linux commands as they are provided in this document.
Do NOT cut them in half, do NOT press enter after part of the command.
WARNING - it is UTTERLY important to copy-paste ENTIRE command, in a SINGLE line, as it is in this document. To help you with that I will post only one command at once. A typical command looks like this:
sudo ufw limit ssh
And will be colored here like a Matrix text for ease of use. Again, even if command is so long that it spans several lines - it still MUST be entered in a single continuous line. 

Laptop requirements

Your computer must have some initial set up for the best experience. The following applications are required:
  • HaasTradeServer - you should have already bought the license and downloaded the file from the profile page
  • Web browser - any will do. You might want to use FireFox or Chrome for the best experience.
  • Remote login program - to connect to remote Linux machine you will need a special application. For the sake of simplicity, I recommend downloading and using free, cross-platform (works on any OS) program Termius
  • File uploader\downloader - to get some files to or from your HTSserver you need a special program. For sake of simplicity, I recommend using cross-platform free program CyberDuck

Pre-install phase - A.K.A. "Ordering and Preparing"

Estimated time ~15 minutes (plus extra for KYC process at hosting of your choice)

These are the steps to make you ready (obviously, you should have downloaded and installed Termius, Cyberduck and HaasTradeServer by now)

  1. If you haven't downloaded HTS yet go to https://www.haasonline.com/account/downloads/ using your Haaslogin and download the "Haas Trade Server" package. Also, save License key somewhere convenient. Open the archive and copy the "linux32.tar.gz" file to easily accessible location.
  2. Register your hosting account, and pick any 4+ CPU cores server. Basically, any hosting provider will do.
  3. If you want fast, reliable and cheap hosting I suggest picking Hetzner. Register your Hosting account and choose any server from the "server auction" https://www.hetzner.com/sb keeping in mind the following:
    • CPU-B rating should be of 7000+
    • Price: should be no more than 33 EUR per month (tax included) and NO setup fee
    • DC: must NOT start with "HEL". This means "Helsinki" location, it is not a good option currently for any of the Asian exchanges (Binance, OKEX and others)
    • A nice additional feature might be iNIC card; it is the iNtel 82574 powerful network adapter. But it is NOT necessary.
    • This - is just perfect
    • IMPORTANT: do NOT look for any "cloud", "SX", "EX", "PX" and other servers. Get one from the "Auction" list.

  4. You must complete some kind of KYC to approve your account. Then wait for the server order to be completed. Note your HTSserver IPv4 address. You will need it.
  5. When you get an email that your server is delivered - use your Hostinglogin, get into hosting ROBOT account (it will be automatically created for you at Hetzner),
    select your server and go to "Linux" tab, select "Ubuntu 18.04 LTS minimal",
    set checkbox "I am aware that blah-blah" and click "Activate installation"
  6. Please note that YOU MUST after that click "Reset" tab, select "Execute an automatic hardware reset" and click "Send" to reboot the server in to initiate OS install.
  7. In a several MINUTES installation will be finished and you will receive an email with the details and information of successful Linux installation.
    Copy and save from there the following fields: IPv4 Address - this is your Server's IP address; Username - this is (most probably "root") your SSHlogin, and Password - this is your SSHpassword.
    If this step is taking more than 10 minutes please make sure you have successfully completed previous "Activate" and "Reboot" steps.
  8. Open Termius (should be already installed from the Termius site) and click "New Host" button.
    Enter your Server IPv4 Address into "Address" field,
    your SSHlogin into "Username" field,
    and SSHPassword into "Password" field. Click save, done!
  9. Open Cyberduck (of course it should already be installed from the Cyberduck site). Open "Bookmarks" in the top menu, pick "New Bookmark",
    select "SFTP", make up some cool Nickname,
    enter your server IPv4 as "address",
    your SSHlogin as "Username"
    and /opt as the path
    Now we are ready for some serious business!
    Make sure everything is in place (SSHpassword will be added later) and proceed (just close this window for MacOS).
  10. Now double-click on your newly created bookmark. At first, you will get a message about server "fingerprint". Don't worry, a new server will always do that. (You should be worried if the same message pops out later, though! You server most probably has been re-set or hacked)
    After that message another one will pop up, asking for the SSHpassword. Enter it here and don't forget to click "save" (or "add to keychain") checkmark to let system remember the password
  11. Copying file: Now you just need to drag-n-drop the HaasTradeServer file (namely the "linux32.tar.gz") to the Cyberduck window. This might take MINUTES. After you confirmed upload completed successfully (no "error" or "canceled" or any other messages) it is safe to exit Cybeduck and proceed to the next step
  12. Now run Termius and double-click on Host we have added previously. You might see another message about "Server Fingerprint", (similar to the one you've seen with CyberDuck already),

    and after accepting those - we are in!

Linux phase - AKA "The Dreadful Mono"

Estimated time ~30 minutes

  1. Updating packages.
    First things first, to keep the Ubuntu happy and fresh we should check for updates. Run the following command:
    sudo apt update
    Enter CASE-SENSITIVE letter Y to any questions.

    If you use up-to-date Ubuntu distributive chances are nothing will be updated.
  2. Installing Mono key
    Now we MUST first switch to Mono repository to get the latest version (it should be 5.xx.xx). Otherwise version 4.6.2 will be installed, which I had some problems with.
    Adding the repository key (it is a SINGLE LINE, do copy the command ENTIRELY):
    sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF The output from the command MUST be error-free.
  3. HTTPS-transport
    To aid with HTTS connectivity it is wise to install apt-https-transport package. Let's do that:
    sudo apt install apt-transport-https This should give no troubles and installs in seconds.
  4. Adding Mono Repository - UBUNTU 18 ONLY
    Now, this is a bit tricky one. You MUST know your Ubuntu version because commands are different. This is a manual for Ubuntu 18. For any other version please consult this page.
    echo "deb https://download.mono-project.com/repo/ubuntu stable-bionic main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.listAlso please mind this is a SINGLE LINE, ONE COMMAND.

    There should be no output for that command.
  5. Pre-Mono steps
    To ensure latest Mono is installed these additional steps must be performed.
    sudo apt update
    After that message indicating "packages might be upgraded" most probably will be show, so we do just that:
    sudo apt upgradeThen some packages will be upgraded; on any question just answer Y
  6. Now the actual MONO installation
    This step will take ~15-30 minutes, depending on your server configuration.
    sudo apt install mono-complete
    As this step is taking a lot of disk space and time you will be asked for confirmation. Just press capital Y to proceed.

    After pretty long time and a lot of text, everything is set
  7. Now we need to install Visual Basic for Mono so newest addition to the family - the Email Bot would work succesfully.
    sudo apt install mono-vbnc
    Of course system will ask you once again if you are completely sure, press Y

    Should take no more than several seconds
  8. Testing Mono installation.
    To do so we must run the command
    mono -VThe result should be basically any version higher than 5.12

    And now to check if we have necessary part of Mono installed, too - the mono-service - run mono-service
    The result must be like "You must specify at least the assembly name" but in general anything other than "No command found" error.

    If you have an older version of Mono or no Service reported this means you have missed some of the steps. Most probably steps to add "Mono key" or "Mono repository".
    It is crucial to go back and re-do those.

Linux final phase - AKA "The HaasTradeServer part"

Estimated time ~7 minutes
  1. Archive checkup.
    Let's go to the /opt folder now
    cd /opt
    And check if our Haas archived file is there
    ls
    We should see the linux32.tar.gz lying there silently

    If the file is not there please go back to "Cyberduck" steps in the beginning and re-do it again.
  2. Folder creation and unpacking.
    Now we need to make the folder for Haas server application.
    mkdir haas
    is a pretty simple command, so we accompany it with unpacking
    tar -C haas -xvf linux32.tar.gz

    Unpacker will show us a list of files it processed - no error should be given!
  3. Pre-configuration run.
    Now we must run Haas once to generate config files. This is easy, change folder
    cd haas
    And run initial
    sudo ./Haasbot.sh

    It would generate a lot of noise for sure, just ignore it for now.
  4. Configuring HaasTradeServer
    Now it's time to perform Haas configuration. Open config in in-built editor
    nano /root/HTS/Settings/MainSettings.XMLThis is how initial config look:

    Now imagine you are using Word (or a Notepad) and change TWO settings. Change ONLY the values. Do NOT delete\remove parameter names (words inside brackets).
    In parameter <OpenInterfaceOnStartup>true</OpenInterfaceOnStartup> change "true" to "false". Resulting line should read: <OpenInterfaceOnStartup>false</OpenInterfaceOnStartup> In parameter <HostingAdres>127.0.0.1</HostingAdres> change the address from 127.0.0.1 to your htsserver IPv4 address. File should be looking a bit like this (of course IP address must be of your server):

    After these two modifications were made, exit editor, saving changes: press CTRL+X, press Y and then press Enter
  5. Finally - the bot launch.
    Now make sure you are in the correct folder (just in case) by running
    cd /opt/haas
    And then, finally, the almighty
    sudo ./Haasbot.sh

    This might look like a lot of errors, but in fact it means literally nothing. We will proceed and check if everything is OK in the next step.
  6. Checking for the bot.Let's see if everything is good. HTS takes a minute to start, so first of all - take small pause (literally minute will be enough). Then run the following
    netstat -tulpn
    You should see your htsserver IP followed by :8090 and another line with :8092

    This indicates successful run.