Build Linux doesn't work


#1

I challenge anyone to install HiFi on a linux ubuntu E2 instance using the official Linux Build Document below:

It just doesn’t work, how can I go about getting it fixed?


#2

Building it as we speak on a fresh VM, 512MB of ram, dual core, with a 30GB hard drive. More or less an EC2 cheapo in a nutshell on Ubuntu 16.04 (I will test 18.04 later on).

Anyway, your argument only had found 2 faults, one of which I pointed out last year:

The other is the python requirement. That one needs discussion from staff, as I have been using 2.7 without any issues. Maybe if you were building the client, I could see a reason for it being an issue, but for the server, it doesn’t seem to matter.

Granted, these are the two only notable issues in the current build guide that I have found so far from testing. I’m not sure which version of Ubuntu is used on EC2 servers at this time (assuming that’s what you meant by E2 instance), so I’ll go ahead and run tests on both, though the build instructions shouldn’t have changed since the update I gave all that while ago except for that python requirement.

EDIT: Decided to just resolve the cmake issue myself:


#3

Okay, culprit #2: VCPKG.

I hate that thing. Anyway, I’m testing out how much memory is needed to get around it, since you only have to run it once (thank god) but like compiling Qt, it takes forever.

Memory Amount Status
512MB Fail
512MB + 512MB Swap Fail
512MB + 1.5GB Swap Testing

Why use swap? Because upping server costs just to recompile the server if the vcpkg packages change is really dumb. My current server is a Digital Ocean server with the happy medium 2 CPU/2GB of ram, where I have a 1GB swap file just in case, so I think 2GB is all that is needed (which is about how much you need to recompile Qt, so that adds up).

Part of why I think VCPKG is a royal pain is because it’s being an a*****e and uses more build processes than what is actually available (I think it’s doing a -j4 if it was a make command). For lower end servers, this is a death wish. Not to mention VCPKG just takes forever.


#4

I first tried it with 18.04 as it says up the top of the official guide. then I found this:

and tried that with 14.04.

Neither of them worked with the problems that you mentioned.

I will try your solution for fixing the cmake problem, thank you for pointing it out. Now I only need to find out how to fix the python problem so I can finally get this F%$#er working.

The last time I used Linux was well over 25 years ago so it’s safe to assume that I know jack S#!t when it comes to this stuff.

Thank you for your reply.


#5

so what I hear you saying is if I change my swap file settings on aws to 2Gig, it should work?


#6

Yep, just chiming back in. Overslept.

Anyway, yes. To appeal to VCPGK, just make sure you have at least 2GB of available memory, swap or otherwise.

fallocate -l 2G swapfile
mkswap swapfile
sudo swapon swapfile

After all is done, just kill the swap file, or not, since it shouldn’t be really used afterwards anyway in normal operation.

As for 14.04 support, I’m really not sure if you can get that working.

Assuming you don’t intend to build interface, I’ve had no issues compiling the server components with python 2.7, so you can get away with just regular python’s repo.

sudo apt install python

#7

ok, I’ll give that a try on 18.04 in a bit and let you know. Thanks again.


#8

fallocate: cannot open swapfile: Permission denied


#9

you need to use sudo

check this tutorial here


#10

We appreciate your feedback, Tim – there are a couple Linux items on the Roadmap right now you can upvote. We agree an official Linux build is a good idea, but we’re not able to commit to a timeline right now.


#11

@Vinny.Mold @FlameSoulis

thanks for the info Vinny, I got further than I have before but still getting errors:

error message

CMake Error at scripts/cmake/vcpkg_download_distfile.cmake:91 (message):

File does not have expected hash:

      File path: [ /tmp/hifi/vcpkg/c1966b8c/downloads/temp/google-draco-1.3.3.tar.gz ]
  Expected hash: [ 80ed5a623046822f5bb26b2454c8ee8cc93ffe9eb3012e8461cefdfc577b26d69a92ea0f0c5e14f5f48c1ef99f9a7263b01710df376792e74358ae14e49c3897 ]
    Actual hash: [ 545253c929d4043149b9bc8a34cbd61631cb8d31ed1fcdfac45f925bd6f187f37113d1f589d4dbcaac93e3f2e61e7769814f634873790e6b17e8dcb035c8c596 ]

The file may have been corrupted in transit. This can be caused by
proxies. If you use a proxy, please set the HTTPS_PROXY and HTTP_PROXY
environment variables to
https://user:password@your-proxy-ip-address:port/”.

Call Stack (most recent call first):
scripts/cmake/vcpkg_download_distfile.cmake:163 (test_hash)
scripts/cmake/vcpkg_from_github.cmake:108 (vcpkg_download_distfile)
ports/draco/portfile.cmake:20 (vcpkg_from_github)
scripts/ports.cmake:71 (include)

Error: Building package draco:x64-linux failed with: BUILD_FAILED
Please ensure you’re using the latest portfiles with .\vcpkg update, then
submit an issue at https://github.com/Microsoft/vcpkg/issues including:
Package: draco:x64-linux
Vcpkg version: 2018.10.20-unknownhash

Additionally, attach any relevant sections from the log files above.

Traceback (most recent call last):
File “/home/ubuntu/hifi/prebuild.py”, line 94, in
main()
File “/home/ubuntu/hifi/prebuild.py”, line 84, in main
pm.setupDependencies()
File “/home/ubuntu/hifi/hifi_vcpkg.py”, line 173, in setupDependencies
self.run([‘install’, ‘–triplet’, self.triplet, ‘hifi-client-deps’])
File “/home/ubuntu/hifi/hifi_vcpkg.py”, line 158, in run
hifi_utils.executeSubprocess(actualCommands, folder=self.path)
File “/home/ubuntu/hifi/hifi_utils.py”, line 61, in executeSubprocess
’ '.join(processArgs[1:]),
RuntimeError: Call to “/tmp/hifi/vcpkg/c1966b8c/vcpkg” failed.

arguments:
–vcpkg-root /tmp/hifi/vcpkg/c1966b8c install --triplet x64-linux hifi-client-deps

CMake Error at CMakeLists.txt:30 (message):
vcpkg configuration missing.

– Configuring incomplete, errors occurred!

Suggestions?


#12

Okay, so can confirm that 2GB is needed. Using swap though is absolutely painfully slow and I do not recommend it at all. If you plan to build your own executables, either do it on your own machine and send it to the VM or upgrade the VM.

I ran it fine in just my home folder.

As for the error… that one is one I never ran into before. Granted, before I used digital ocean servers, I ran tests on amazon EC2 servers, so that’s why I’m a little confused. The most I can suggest is clearing /tmp/hifi and trying it again. Also, VCPKG takes up a good chunk of space, but with the OS and everything else, I only used less than 9GB on my current VM for testing.


#13

This is the error that I’m getting with the build guide and 2gb of swap; i may be forgetting the environment variables.

Elapsed time for package hifi-client-deps:x64-linux: 9.42 ms

Total elapsed time: 4.848 min

Wiping build trees
Writing cmake config to /home/mold/hifi/build/vcpkg.cmake
CMake Error at CMakeLists.txt:24 (if):
  if given arguments:

    "CMAKE_VERSION" "VERSION_GREATER_EQUAL" "3.12"

  Unknown arguments specified


-- Configuring incomplete, errors occurred! 

I am using cmake 3.5.1

UPDATE: Follow the instructions in the second answer to update cmake in 16.04 https://askubuntu.com/questions/355565/how-do-i-install-the-latest-version-of-cmake-from-the-command-line/595441#595441


#14

The heck? Since when did they put that in? They recommend Cmake 3.9. In fact, I used my own notes for the 3.9 fix to do it so I’m confused. That being said, I am glad it has some kind of CMAKE check (different issue on Github).


#15

Here’s my logs when updated to cmake version 3.13.4

cmake -DQT_CMAKE_PREFIX_PATH=/usr/local/Qt5.10.1/5.10.1/gcc_64/lib/cmake -DSERVER_ONLY=1  ..
['/home/mold/hifi/prebuild.py', '--build-root', '/home/mold/hifi/build']
Warning: Environment variable HIFI_VCPKG_BASE not set, using /tmp/hifi/vcpkg
Using vcpkg path /tmp/hifi/vcpkg/c1966b8c
Writing tag c1966b8c_1 to /tmp/hifi/vcpkg/c1966b8c/.id
Installing host tools
Running command
['/tmp/hifi/vcpkg/c1966b8c/vcpkg', '--vcpkg-root', '/tmp/hifi/vcpkg/c1966b8c', 'install', '--triplet', 'x64-linux', 'hifi-host-tools']
The following packages are already installed:
    hifi-host-tools[core]:x64-linux
Starting package 1/1: hifi-host-tools:x64-linux
Package hifi-host-tools:x64-linux is already installed
Elapsed time for package hifi-host-tools:x64-linux: 5.16 us

Total elapsed time: 10.33 us

Installing build dependencies
Running command
['/tmp/hifi/vcpkg/c1966b8c/vcpkg', '--vcpkg-root', '/tmp/hifi/vcpkg/c1966b8c', 'install', '--triplet', 'x64-linux', 'hifi-client-deps']
The following packages are already installed:
    hifi-client-deps[core]:x64-linux
Starting package 1/1: hifi-client-deps:x64-linux
Package hifi-client-deps:x64-linux is already installed
Elapsed time for package hifi-client-deps:x64-linux: 5.185 us

Total elapsed time: 9.636 us

Writing cmake config to /home/mold/hifi/build/vcpkg.cmake
rm: cannot remove '/tmp/vcpkg/vcpkg0b137a96-ee43-467f-8e78-913403097982.txt': No such file or directory
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test COMPILER_SUPPORTS_CXX14
-- Performing Test COMPILER_SUPPORTS_CXX14 - Success
-- Build server:          ON
-- Build client:          OFF
-- Build tests:           OFF
-- Build tools:           ON
-- Build installer:       ON
-- GL ES:                 OFF
-- DL serverless content: OFF
-- The Qt build in use is: "/usr/local/Qt5.10.1/5.10.1/gcc_64"
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- The RELEASE_TYPE variable is: PRODUCTIONBUILD_GLOBAL_SERVICES=STABLE
CMake Error at cmake/macros/SetPackagingParameters.cmake:86 (message):
  Cannot produce STABLE_BUILD without PRODUCTION_BUILD
Call Stack (most recent call first):
  CMakeLists.txt:175 (set_packaging_parameters)


-- Configuring incomplete, errors occurred!
See also "/home/mold/hifi/build/CMakeFiles/CMakeOutput.log".
See also "/home/mold/hifi/build/CMakeFiles/CMakeError.log".

Output of CMakeError.log

Determining if the pthread_create exist failed with the following output:
Change Dir: /home/mold/hifi/build/CMakeFiles/CMakeTmp

Run Build Command:"/usr/bin/make" "cmTC_71eee/fast"
/usr/bin/make -f CMakeFiles/cmTC_71eee.dir/build.make CMakeFiles/cmTC_71eee.dir/build
make[1]: Entering directory '/home/mold/hifi/build/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_71eee.dir/CheckSymbolExists.c.o
/usr/bin/cc    -o CMakeFiles/cmTC_71eee.dir/CheckSymbolExists.c.o   -c /home/mold/hifi/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c
Linking C executable cmTC_71eee
/opt/cmake/cmake-3.13.4-Linux-x86_64/bin/cmake -E cmake_link_script CMakeFiles/cmTC_71eee.dir/link.txt --verbose=1
/usr/bin/cc      CMakeFiles/cmTC_71eee.dir/CheckSymbolExists.c.o  -o cmTC_71eee
CMakeFiles/cmTC_71eee.dir/CheckSymbolExists.c.o: In function `main':
CheckSymbolExists.c:(.text+0x16): undefined reference to `pthread_create'
collect2: error: ld returned 1 exit status
CMakeFiles/cmTC_71eee.dir/build.make:86: recipe for target 'cmTC_71eee' failed
make[1]: *** [cmTC_71eee] Error 1
make[1]: Leaving directory '/home/mold/hifi/build/CMakeFiles/CMakeTmp'
Makefile:121: recipe for target 'cmTC_71eee/fast' failed
make: *** [cmTC_71eee/fast] Error 2

File /home/mold/hifi/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c:
/* */
#include <pthread.h>

int main(int argc, char** argv)
{
  (void)argv;
#ifndef pthread_create
  return ((int*)(&pthread_create))[argc];
#else
  (void)argc;
  return 0;
#endif
}

Determining if the function pthread_create exists in the pthreads failed with the following output:
Change Dir: /home/mold/hifi/build/CMakeFiles/CMakeTmp

Run Build Command:"/usr/bin/make" "cmTC_43060/fast"
/usr/bin/make -f CMakeFiles/cmTC_43060.dir/build.make CMakeFiles/cmTC_43060.dir/build
make[1]: Entering directory '/home/mold/hifi/build/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_43060.dir/CheckFunctionExists.c.o
/usr/bin/cc   -DCHECK_FUNCTION_EXISTS=pthread_create   -o CMakeFiles/cmTC_43060.dir/CheckFunctionExists.c.o   -c /opt/cmake/cmake-3.13.4-Linux-x86_64/share/cmake-3.13/Modules/CheckFunctionExists.c
Linking C executable cmTC_43060
/opt/cmake/cmake-3.13.4-Linux-x86_64/bin/cmake -E cmake_link_script CMakeFiles/cmTC_43060.dir/link.txt --verbose=1
/usr/bin/cc  -DCHECK_FUNCTION_EXISTS=pthread_create    CMakeFiles/cmTC_43060.dir/CheckFunctionExists.c.o  -o cmTC_43060 -lpthreads
/usr/bin/ld: cannot find -lpthreads
collect2: error: ld returned 1 exit status
CMakeFiles/cmTC_43060.dir/build.make:86: recipe for target 'cmTC_43060' failed
make[1]: *** [cmTC_43060] Error 1
make[1]: Leaving directory '/home/mold/hifi/build/CMakeFiles/CMakeTmp'
Makefile:121: recipe for target 'cmTC_43060/fast' failed
make: *** [cmTC_43060/fast] Error 2

The output of CMakeOutput.log does not appear to be relevant, and CMakeOutput is long but can be included if needed

EDIT: I clearly missed the main point here in the log: “Cannot produce STABLE_BUILD without PRODUCTION_BUILD”. Now to figure out what this means.

EDIT2: Earlier, i used “export STABLE_BUILD=1” and had to set it to 0 in order for hifi to build. I guess those environment variables from 78 are not functional for 79?

EDIT3: It’s building now. I guess we’ll have to see if I’m actually able to connect to it later.


#16

well, it looks like it’s currently building the domain server and assignment client. I didn’t worry about the interface. I’ll be happier when it’s finished but it’s looking good. thank you flame and vinnie for your help. I literally couldn’t have done it without your help.

One more question if I may, I noticed that it said dev build as it was setting up, how do I go about changing this to production build, assuming this is what I’ll need to host my own domain?


#17

ok, it installed both domain server and assignment client and I managed to get them both going.

I’m having trouble connecting to the port 40100. I’ll have to find out how to open a specific port in AWS.