Currently Viewing Posts Tagged ubuntu

Migrating GitLab to another Server from Ubuntu 16.04 to Ubuntu 18.04

Because my old Gitlab is running on a Ubuntu 16.04 server. The latest LTS version of Ubuntu is already 18.04 for more than one year. I was thinking about to directly upgrade the server from 16.04 to 18.04.  There are still some chances it may fail and the server may crash. I can not afford to lose the data.

So the final decision is to migrate the Gitlab from the old server Ubuntu 16.04 to a new server with  Ubuntu 18.04.

This is how to do it.

First, I install the new server on my Proxmox VE environment with Ubuntu 18.04. 3CPU Cores and 4GB memory and 64GB storage. Meanwhile, the old server still uses Ubuntu 16.04. And the Gitlab version that I use is 12.4.2

On the Old Server

  1.  Shutdown Gitlab service
    sudo gitlab-ctl stop unicorn
    sudo gitlab-ctl stop sidekiq
  2. Backup Gitlab on the old server
    sudo gitlab-rake gitlab:bakcup:create
  3. Create a folder named gitlab-old
    mkdir ~/gitlab-old
  4. Copy the backup file to the folder ~/gitlab-old
    sudo cp ~/backups/xxxxxxxx_gitlab_backup.tar ~/gitlab-old
  5. Copy the Gitlab configuration files
    sudo cp /etc/gitlab/gitlab.rb ~/gitlab-old
    sudo cp /etc/gitlab/gitlab-secrets.json ~/gitlab-old
    sudo cp -R /etc/gitlab/ssl ~/gitlab-old

Then all the important files are under folder ~/gitlab-old

My configuration of Gitlab changed the backup folder to ~/backups. The default path is /var/opt/gitlab/backups/

You can change it on the “Backup Settings” section in the file gitlab.rb

 

To verify the version of your gitlab installation, enter the following command.

sudo gitlab-rake gitlab:env:info

On the New Server

The server is a clean installation of Ubuntu 18.04. The only service is SSH  and postfix.

Before doing the migration, I transfer the whole folder ~/gitlab-old from the old server to this new server.

  1.  Add Gitlab source
    curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash
  2.  Install Gitlab 12.4.2 CE
    sudo apt update
    sudo apt install gitlab-ce=12.4.2
  3. Copy the configuration files to folder /etc/gitlab
    sudo cp ~/gitlab-old/gitlab* /etc/gitlab
  4. Copy the ssl folder to folder /etc/gitlab
    sudo cp -R ~/gitlab-old/ssl /etc/gitlab
  5. Run Gitlab service for the first time
    sudo gitlab-ctl reconfigure
  6. Shutdown the Gitlab services
    sudo gitlab-ctl stop unicorn
    sudo gitlab-ctl stop sidekiq
  7. Copy the backup file to ~/backups then change the permission
    sudo cp ~/gitlab-old/*.tar ~/backups
    sudo chown git:root ~/backups/*.tar
  8. Restore the Gitlab backup file
    sudo gitlab-rake gitlab:backup:restore BACKUP=XXXXXX
  9. Restart Gitlab and check
    sudo gitlab-ctl start
    sudo gitlab-rake gitlab:check SANITIZE=true

There are some notes for the above nine steps.

The backup file name looks like 1573175514_2019_11_07_12.4.2_gitlab_backup.tar. So the code in the step 8, BACKUP=1573175514_2019_11_07_12.4.2

The version of Gitlab I am using 12.4.2. I install the 12.4.2 on step 2. When I do the restore in step 8, it displays the error, the version does not match. The backup is 12.4.2, but the installation is 12.4.1. I have to do the apt update and apt upgrade to upgrade the Gitlab from 12.4.1 to 12.4.2. I am not sure where the problem came from. Just a reminder, double-check the version installed on the new server.

At the first step of adding the repo source, I always get the network error on my server. I believed it is the problem of guest OS on Proxmox VE. So I actually add the repo source and public key of gitlab source manually.

My current Gitlab information on the new server Ubuntu 18.04

davidyin@gitlab:~$ sudo gitlab-rake gitlab:env:info

System information
System: Ubuntu 18.04
Current User: git
Using RVM: no
Ruby Version: 2.6.3p62
Gem Version: 2.7.9
Bundler Version:1.17.3
Rake Version: 12.3.3
Redis Version: 3.2.12
Git Version: 2.22.0
Sidekiq Version:5.2.7
Go Version: unknown

GitLab information
Version: 12.4.2
Revision: 393a5bdafa2
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 10.9
URL: https://gitlab.g2soft.net
HTTP Clone URL: https://gitlab.g2soft.net/some-group/some-project.git
SSH Clone URL: git@gitlab.g2soft.net:some-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version: 10.2.0
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Git: /opt/gitlab/embedded/bin/git

At the end, I should give the credit to Didiet A. Pambudiono. Most of the steps above is based on this post.

Force IPv4 when using APT command in Ubuntu

Sometimes you may want to disable IPv6 when using apt update command or some other apt operation.

If you want to use IPv4 in one command, just run following:

$ sudo apt -o Acquire::ForceIPv4=true update

If you want to force to use IPv4 for all apt operations, add a new file under /etc/apt/apt.conf.d/

$ sudo nano /etc/apt/apt.conf.d/99force-ipv4

Then add a line:

Acquire::ForceIPv4 "true";

If you want to disable IPv4, change the IPv4 to IPv6.

Move Virtualbox VDI to Proxmox VE System

I have some Virtualbox virtual machine on my desktop. Now I have Proxmox VE 5.3-8 installed on a separate machine. So I would like to move the VM to Proxmox.

Let me list the existing installation.

  • My Desktop, Windows 10 Pro version 1809.
  • Virtualbox 6.0.4 r128413
  • Guest VM inside of Virtualbox is Ubuntu 18.04 with 1G memory and 20GB disk space.
  • Proxmox VE 5.3-8, VM and LXC are stored in the local-lvm

 

Step 1, Located the vdi file and convert it to RAW format

Virtualbox vdi file location

Check the settings of the virtual machine, storage, and located the location of the vdi file. Mine is in the d:\Virtualbox VM\

Then open a command prompt window, enter the following command to convert the vdi file to raw image file.

d:
cd "program files\oracle\virtualbox"
vboxmanage clonehd --format RAW "d"\virtualbox vm\ubuntu 18.04 (122)\ubuntu 18.04.vdi" d:\testing.img

After a while the new image file is ready. The time depends on the size of your VM. Maybe 10 minutes or half an hour.

Converting

Continue reading “Move Virtualbox VDI to Proxmox VE System”

How to take a webpage screenshot at terminal window

The scenario is to run the program inside of Ubuntu VPS.  Input a URL the program will output the jpg file.

The solution is to use wkhtmltoimage, which is a part of wkhtmtopdf.

I used to use the old version wkhtmltopdf v0.12.3. Now it is not working anymore due to some reason I don’t know.

But the wkhtmltopdf v0.12.5  is OK.

The OS is Ubuntu 18.04.

Then I install wkhtmltopdf with the following command

sudo apt-get install -y software-properties-common
sudo apt-add-repository -y "deb http://security.ubuntu.com/ubuntu xenial-security main" 
sudo apt-get -yq update
sudo apt-get install -y libxrender1 libfontconfig1 libx11-dev libjpeg62 libxtst6 fontconfig xfonts-75dpi xfonts-base libpng12-0 libjpeg-turbo8
wget "https://downloads.wkhtmltopdf.org/0.12/0.12.5/wkhtmltox_0.12.5-1.xenial_amd64.deb" 
sudo dpkg -i wkhtmltox_0.12.5-1.xenial_amd64.deb 
sudo apt-get -f install

After installation, run the command to show the help of the program.

Continue reading “How to take a webpage screenshot at terminal window”

Use GoAccess to Generate Report with Multiple Nginx Log Files

I have a VPS with Nginx as a web server and also a lot of web sites on it. It is on Digital Ocean.

Here is the VPS basic information.

  • SFO2
  • 1GB Nanode
  • Ubuntu Linux 18.04.2
  • Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz, 1 cores
  • Nginx 1.15.9

All sites use one access.log file. The following code is in the http block of /etc/nginx/nginx.conf file.

log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';

access_log /var/log/nginx/access.log main;

All access logs look like below:

Now it is time to show you how to use GoAccess to Generate Report with Multiple Nginx Log files.

Install GoAccess

sudo install goaccess

 

Modify the Config file of GoAccess

GoAccess config file is located at /etc/

sudo nano /etc/goaccess.conf

Add the following code or modify the code inside of the goaccess.conf.

time-format %H:%M:%S

date-format %d/%b/%Y

log-format %h %^[%d:%t %^] "%r" %s %b "%R" "%u"

Run GoAccess to generate the report

I would like to get a static report.

sudo goaccess /var/log/nginx/access.log -o /home/davidyin/mywebsite.com/report.html --log-format=COMBINED

The report looks like below:

Due to the log is not just one file. The log files are generated by date and also compressed.

Two log files are not compressed.

  • access.log
  • access.log.1

The older log files are compressed.

  • access.log.2.gz
  • access.log.3.gz
  • access.log.4.gz
  • access.log.5.gz

I want to generate all the access log files. So the command is changed to the way which can phrase multiple log files.

zcat /var/log/nginx/access.log.*.gz | goaccess /var/log/nginx/access.log /var/log/nginx/access.log.1 - -o /home/davidyin/mywebsite.com/report.html --log-format=COMBINED

The sample command above using mywebsite.com. Please replace it with your own sites.

At the last, I add this command into Cron jobs. Schedule it every hour.

Ubuntu 17.10 (Artful Aardvark) End of Life reached on July 19 2018

Today, I received the email, regarding the life of Ubuntu 17.10. You will see it is end of life on July 19, 2018, means Today.
If you are still using Ubuntu 17.10, you would better to upgrade to Ubuntu 18.04.

This is a follow-up to the End of Life warning sent earlier this month
to confirm that as of today (July 19, 2018), Ubuntu 17.10 is no longer
supported. No more package updates will be accepted to 17.10, and
it will be archived to old-releases.ubuntu.com in the coming weeks.

The original End of Life warning follows, with upgrade instructions:

Ubuntu announced its 17.10 (Artful Aardvark) release almost 9 months
ago, on October 19, 2017. As a non-LTS release, 17.10 has a 9-month
support cycle and, as such, the support period is now nearing its
end and Ubuntu 17.10 will reach end of life on Thursday, July 19th.

At that time, Ubuntu Security Notices will no longer include
information or updated packages for Ubuntu 17.10.

The supported upgrade path from Ubuntu 17.10 is via Ubuntu 18.04.
Instructions and caveats for the upgrade may be found at:

https://help.ubuntu.com/community/BionicUpgrades

Ubuntu 18.04 continues to be actively supported with security updates
and select high-impact bug fixes. Announcements of security updates
for Ubuntu releases are sent to the ubuntu-security-announce mailing
list, information about which may be found at:

https://lists.ubuntu.com/mailman/listinfo/ubuntu-security-announce

Since its launch in October 2004 Ubuntu has become one of the most
highly regarded Linux distributions with millions of users in homes,
schools, businesses and governments around the world. Ubuntu is Open
Source software, costs nothing to download, and users are free to
customise or alter their software in order to meet their needs.

On behalf of the Ubuntu Release Team,

Adam Conrad

Waiting for the Ubuntu 18.04 LTS

Based on the schedule of release, Ubuntu 18.04 LTS (Bionic Beaver) will be released on April 26, 2018, the day after tomorrow.

Now, my all VPS is running on the Ubuntu 16.04 LTS. LTS is an abbreviation for “Long Term Support”.  An LTS version is released every two years. So after Ubuntu 16.04 LTS, it is Ubuntu 18.04 LTS.  Now all LTS version has five years support.

I can use Ubuntu 16.04 LTS until 2021. Security updates will be released during the time period.

I am waiting for the release of Ubuntu 18.04 LTS. I would like to try it on my staging server, or a virtual machine. Try all my existing web programs on Ubuntu 18.04 LTS. Find the compatible problems then fix them.

Maybe after a few months testing. I am sure there are no issues with it. I will install a new VPS with Ubuntu 18.04 LTS and move the websites to it.

Continue reading “Waiting for the Ubuntu 18.04 LTS”

  • Archives