Support multiple mDNS backends

It would be great if one can choose between the new internal and the previously supported Avahi mDNS backend.
The reason is that Avahi is widely spread in the Linux universe. Using the internal backend adds redundant functionality and furthermore increases the binary size which is undesirable for embedded targets.

I'd rather have an option to disable discovery than supporting different backends.
Multiple audio backends are already complicated enough.
By removing discovery altogether, you can also get rid of hyper, url, idna, cookie, and probably a few other small dependency. These account for quite a bit :
https://github.com/plietar/librespot/issues/91#issuecomment-233194833
librespot saves credentials (just a token, not the password) to disk, so even without discovery you only need to login once.

I really like the discovery feature as it requires zero user configuration. This is my naive approach to support both backends: https://github.com/joerg-krause/librespot/commit/bd2ad97c068d4bad7dc52d1fd87d8f834393c289.
I know both features should be exclusive, but this is beyond my abilities for now.

FWIW, I compared the binary sizes for building librespot with the internal and the avahi backend:
internal:
   text	   data	    bss	    dec	    hex	filename
2638364	 278916	    860	2918140	 2c86fc	target/armv5te-unknown-linux-musl/release/librespot

avahi:
   text	   data	    bss	    dec	    hex	filename
2497189	 277504	    852	2775545	 2a59f9	target/armv5te-unknown-linux-musl/release/librespot

So, using the avahi-backend saves me around 140kB (avahi is already installed and needed by other processes).
I fully understand that you want to keep the build complexity low. However, as related to #91 it would be great to reduce the binary size especially for embedded devices.

Eventually, I am fine using rust-mdns. So I close this...


Update capture.cpp

Fixes #484.  Fixes #498
Adds camera ID (for USB 3 cameras) and camera serial number to log output.  These will be used in a future update that has different settings for different cameras.

Chris, how do I submit a new commit to PR #499 to fix the issues you identified?

Not sure how to do that with the web ui, but it's easy at the CLI.

~install git~
~check out the source~
~create a branch~ (https://github.com/thomasjacquin/allsky/tree/Camera_ID_serialNumber_plus2fixes)
make changes
test changes
commit changes (git add , git commit)
push changes (git push)
make more changes
push changes (git add, git commit, git push)
...
clean up your local repo (git checkout master, git branch -D whateverbranchyoucreatedforthisparticularchange)

It looks like you have a working branch for this change, so I assume you have git and the allsky source on your Pi. But you don't need to work on your Pi - you could do this inside a VM on your laptop/desktop. Anyway, you just keep committing and pushing changes on that branch until it's ready to be merged.
Github's docs are pretty good:

https://docs.github.com/en/get-started/quickstart/fork-a-repo
https://docs.github.com/en/github/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-and-deleting-branches-within-your-repository
https://docs.github.com/en/github/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/about-pull-request-reviews
https://docs.github.com/en/github/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/incorporating-feedback-in-your-pull-request
https://docs.github.com/en/github/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests

Maybe there's something in the github docs about making changes using the github Web UI.

@EricClaeys would you be offended if I take over this PR and either try fix it, or close/reject it and make a new PR?

@ckuethe no problem if you want to fix it.  I view us as a team all working toward the same goal, so it's not too important to me who does what.  Thanks for asking.
@linuxkidd please take a look at my newest capture_RPiHQ.cpp prior to refactoring.  It's has many of the same functions as capture.cpp.
Eric

Interesting, The 120MM-S on my roof doesn't seem to support the serial number field.
Sep 29 13:35:56 skycam allsky.sh[3179]: *** WARNING: unable to get serialNumber (16 (ASI_ERROR_GENERAL_ERROR))

That doesn't stop capture from running properly, but I guess we will need to be a little bit careful about assuming that a serial number will be present.

Here’s what the ZWO tech said.  I agree we need to gracefully handle cameras without a serial #:
Almost all cameras support serial number, except for a few very old cameras.
In ASICap, if you can see the SN, that means the camera supports serial number.
From: Chris Kuethe @.>
Sent: Wednesday, September 29, 2021 4:54 PM
To: thomasjacquin/allsky @.>
Cc: EricClaeys @.>; Mention @.>
Subject: Re: [thomasjacquin/allsky] Update capture.cpp to log camera serial number and configurable ID field (#499)
Interesting, The 120MM-S on my roof doesn't seem to support the serial number field.
Sep 29 13:35:56 skycam allsky.sh[3179]: *** WARNING: unable to get serialNumber (16 (ASI_ERROR_GENERAL_ERROR))
That doesn't stop capture from running properly, but I guess we will need to be a little bit careful about assuming that a serial number will be present.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/thomasjacquin/allsky/pull/499#issuecomment-930535378 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AT2PYK2S3LS6OIK7JLXDD43UEN4IBANCNFSM5EUR74AQ .
Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675  or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub .   https://github.com/notifications/beacon/AT2PYKYDZCB7UF6QZCRWOV3UEN4IBA5CNFSM5EUR74A2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOG53NPUQ.gif


🧑💻: Sudoku game in python

:red_circle: Sudoku game :
:red_circle: want to add a python based sudoku game that i have created :
:red_circle: Changes :

creating a sudoku game using backtracking in python

Screenshots 📷

:white_check_mark: To be Mentioned while taking the issue :

Full name : Arindam Bera
What is your participant role? (Mention the Open Source Program name. Eg. GSOC, GSSOC, SSOC, JWOC, etc.)
GSSOC

Happy Contributing 🚀
All the best. Enjoy your open source journey ahead. 😎

@UTSAVS26 hii!! please assign me this issue.. I'll work very well and good in developing this game

Can you please assign me this issue.


sort by key not working for countries method

This is the code snippet i have used for accessing countries method with sort by key
exports.getTest = (req, res) => {
    (async() => {
        sortedCountries = await covid.countries('cases', 'sort by');
        res.send(sortedCountries);
    })();
}

But it returns that country not found.
I think there is some collision between sort and get specific country methods.

sorry for the inconvinience, covid.countries(null, 'cases') worked for me


Segmentation fault when using threading

Linux version:
64-bit Ubuntu

PyQ version:
PyQ 4.1.3
NumPy 1.13.3
KDB+ 3.5 (2018.02.26) l64
Python 2.7.12 (default, Dec  4 2017, 14:50:18)
[GCC 5.4.0 20160609]

kdb+ version:
64-bit kdb+
QLIC=/home/zakariyya/q/q64

Not install via virtual environment

QHOME=/home/zakariyya/q/q64

Not using Conda

Hi guys,
I'm getting a segmentation fault when running a q process that loads in a Python script (.p)
The use-case if fairly simple. I'm exposing Python subscription and unsubscription functions for Redis to q, then I call them so that they can run in a separate thread. That means that q can still work in "parallel"
$ cat redis.q
system "l /home/zakariyya/redis_thread.p";
.z.pc:{ if[x ~ hndl; -2" lost handle to tp"; if[`res in key `.;unsubscribe_redis enlist (res)] ] };
.z.exit:{ if[`res in key `.;unsubscribe_redis enlist (res)]; };
hndl:hopen 6006;
upd:{[t;d] data:d; neg[hndl](`upd;`tbl; data)};
show .z.i
$ q redis.q
KDB+ 3.5 2018.02.26 Copyright (C) 1993-2018 Kx Systems
l64/ 8()core 15999MB zakariyya zakariyya-pc-2193 <IP_ADDRESS> EXPIRE 2018.07.01 zak********* KOD #51578

3645i
q)res:subscribe_redis("quote*";"quote")

And the Python script
$ cat redis_thread.p
import threading
import sys
import time
import redis
import signal
import os
from functools import partial
from pyq import q, K, _k

class PublisherThread(threading.Thread):

    def __init__(self, channel, r, kdb_table):
        super(PublisherThread, self).__init__()
        self._stopper = threading.Event()
        self.channel = channel
        self.kdb_table = kdb_table
        self.redis = r
        self.pubsub = self.redis.pubsub()
        self.redis.client_setname("kdb-feed-" + self.kdb_table)
        self.pubsub.psubscribe(self.channel)

    def stop(self):
        print('Closing Redis connection...')
        self.pubsub.close()
        self._stopper = True

    @property
    def stopped(self):
        return self._stopper.isSet()

    def run(self):
        while not self.stopped:
            try:
                msg = self.pubsub.get_message()
                if msg:
                    if msg['type'] in ('message', 'pmessage'):
                        #print(msg)
                        qmsg = K.string(msg)
                        q('upd', self.kdb_table, qmsg)
                        time.sleep(0.001)
            except _k.error as e:
                print('Caught Q error. Cannot insert data to table')
                self.stop()
            except Exception as e:
                print('Received unhandled exception. Cannot insert data to table')
                self.stop()

class RedisManager(object):

    def __init__(self, subscriber_init):
        self._subscribers_store = {}
        self.subscriber_init = subscriber_init

    def add(self, feed, kdb_table):
       print('Subscribing')
        key = ':'.join([feed, kdb_table])
        self._subscribers_store[key] = self.subscriber_init(feed, kdb_table)
        return key

    def remove(self, key):
        self._subscribers_store[key].stop()
        del self._subscribers_store[key]
        return True

def subscriber_init(feed, kdb_table, redis_client):
    t = PublisherThread(feed, redis_client, kdb_table)
    t.start()
    return t

# Create curried function of RedisManager's init
redis_manager = RedisManager(
    partial(subscriber_init,
         redis_client=redis.StrictRedis(host='XX.XXX.XXX.XXX', port=6399, db=0)))

# Create and expose Python functions a q callables

def q_subscribe_redis(feed, kdb_table):
    return redis_manager.add(str(feed), str(kdb_table))

def q_unsubscribe_redis(key):
    return redis_manager.remove(str(key))

q.subscribe_redis = q_subscribe_redis
q.unsubscribe_redis = q_unsubscribe_redis

And the segmentation fault:
q)Sorry, this application or an associated library has encountered a fatal error and will exit.
If known, please email the steps to reproduce this error to<EMAIL_ADDRESS>with a copy of the kdb+ startup banner.
Thank you.
/home/zakariyya/q/q64/l64/q() [0x47a8b1]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7fc0aa739390]
/home/zakariyya/q/q64/l64/q(r0+0) [0x41bd40]
/home/zakariyya/q/q64/l64/q() [0x408a9c]
/home/zakariyya/q/q64/l64/q() [0x40fb6f]
/home/zakariyya/q/q64/l64/q() [0x4042c8]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7fc0aa37e830]
/home/zakariyya/q/q64/l64/q() [0x4043a1]
rlwrap: warning: q crashed, killed by SIGSEGV (core dumped).
rlwrap itself has not crashed, but for transparency,
it will now kill itself with the same signal

warnings can be silenced by the --no-warnings (-n) option
Segmentation fault (core dumped)

I did not have time to reproduce your issue, but from the backtrace it looks like you are passing a K object from one thread to another resulting in r0 being called in one thread on an object created in another thread.  To avoid this you should convert K objects to native Python objects whenever they can be shared between threads.

@zak-b2c2 - your redis.q scrip attempts to hopen a handle to port 6006.  What process is listening on this port?  An unmodified redis server does not accept kdb+ connections.

Thanks for the missing part, but I still cannot reproduce your problem.  For me, the your redis.q script hangs after printing "Subscribing" and after a few minutes prints a connection error:
$ q redis.q
KDB+ 3.5 2017.06.19 Copyright (C) 1993-2017 Kx Systems
m32/ 20()core 65536MB a **** NONEXPIRE

98033i
q)subscribe_redis("quote*";"quote")
Subscribing
'ConnectionError
  [0]  subscribe_redis("quote*";"quote")
       ^

Note that when you call q('upd', self.kdb_table, msg), msg is implicitly cast to a q symbol, so this is not what you might want to do.
I am running a redis server on the default 6379 port.  How do you run the redis server?

Aha, I missed the
redis_client=redis.StrictRedis(host='XX.XXX.XXX.XXX', port=6399, db=0)

part.  Let me restart redis on port 6399.

Sorry, yes I blurred out the host and port for security reasons. It's an in-house Redis server

OK, let me try <IP_ADDRESS> ...

Now it looks like I am getting a connection but still no crash:
q)subscribe_redis("quote*";"quote")
Subscribing
`quote*:quote
q)

Apparently, there is something on your redis server that causes the problem.  Can you reproduce the issue using a local redis instance with a minimal setup that you can share?

Two things:

The error happens randomly (can be after 10 seconds or after 20 minutes)
The throughput is about 10,000 messages/minute

Ok I've managed to reproduce on a local instance

Redis host = <IP_ADDRESS>
Redis port = 6379

I have 3 q processes with different functionalities:

Process 1 - Redis Publisher

$ q
q).z.ts:{ do[100;system "redis-cli -h <IP_ADDRESS> -p 6379 PUBLISH quote Hello"] }
q)\t 100

Process 2  - Redis Listener

$ q redis.q
KDB+ 3.5 2018.02.26 Copyright (C) 1993-2018 Kx Systems
l64/ 8()core 15999MB zakariyya zakariyya-pc-2193 <IP_ADDRESS> EXPIRE 2018.07.01 zak********* KOD #51578

27814i
q)res:subscribe_redis("quote*";"quote")
Subscribing
q)

Process 3 - "Tickerplant"

$ q
q)tbl:([] msg:());
q)upd:{x insert enlist enlist y};

And the error raised on Process 2
q)Sorry, this application or an associated library has encountered a fatal error and will exit.
If known, please email the steps to reproduce this error to<EMAIL_ADDRESS>with a copy of the kdb+ startup banner.
Thank you.
/home/zakariyya/q/q64/l64/q() [0x47a8b1]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7f8a1de58390]
/home/zakariyya/q/q64/l64/q(r0+0) [0x41bd40]
/home/zakariyya/q/q64/l64/q() [0x408a9c]
/home/zakariyya/q/q64/l64/q() [0x40fb6f]
/home/zakariyya/q/q64/l64/q() [0x4042c8]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f8a1da9d830]
/home/zakariyya/q/q64/l64/q() [0x4043a1]
rlwrap: warning: q crashed, killed by SIGSEGV (core dumped).
rlwrap itself has not crashed, but for transparency,
it will now kill itself with the same signal

warnings can be silenced by the --no-warnings (-n) option
Segmentation fault (core dumped)

Unfortunately, without being able to reproduce your issue, I don't think we can help you.  A few general remarks:

Your redis.q process sends asyncronous updates to a separate kdb+ instance and then sleeps for a fraction of a second.  Why not use synchronous updates?

Do you need to run a Redis client and kdb+ in separate processes? Why not have redis.q process manage the kdb+ tables directly?

Do you need multiple threads?  Take a look at asyncio framework or its Python 2 precursors.

Still no crash.  On the "Tickerplant", I get
q)count tbl
9500
q)count tbl
9817
q)count tbl
9983
q)count tbl
10645
q)count tbl
11032
q)-5#tbl
msg
----------------------------------------------------------------
`type`pattern`channel`data!("pmessage";"quote*";"quote";"Hello")
`type`pattern`channel`data!("pmessage";"quote*";"quote";"Hello")
`type`pattern`channel`data!("pmessage";"quote*";"quote";"Hello")
`type`pattern`channel`data!("pmessage";"quote*";"quote";"Hello")
`type`pattern`channel`data!("pmessage";"quote*";"quote";"Hello")

I'll check if it behaves differently with Python 2.7 and released version of PyQ.  My current setup is
$ pyq --versions
PyQ 4.2.0.dev141+b6bfbfa
NumPy 1.14.0
KDB+ 3.5 (2017.06.19) m32
Python 3.6.4 (default, Jan  8 2018, 14:03:33)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-9<IP_ADDRESS>)]

Same story with
$ pyq --versions
PyQ 4.1.3
NumPy 1.14.2
KDB+ 3.5 (2017.06.19) m32
Python 2.7.14 (default, Jan  8 2018, 14:00:34)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-9<IP_ADDRESS>)]

The "tbl" is happily growing on the "Tickerplant".
q)count tbl
2236
q)count tbl
2400
q)count tbl
2521

Two things:

If you can, could you let it running for a while like 15 minutes or more?
Finally, if you could crank up the load on .z.ts?  .z.ts:{ do[500;system "redis-cli -h <IP_ADDRESS> -p 6379 PUBLISH quote Hello"] }

I let the setup run for a while and after almost half a million messages, got
q) lost handle to tp
Closing Redis connection...
'snd. OS reports: Bad file descriptor
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/local/Cellar/python36/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "<string>", line 24, in run
  File "<string>", line 20, in stopped
AttributeError: 'bool' object has no attribute 'isSet'

There is too much of irrelevant code in your redis_thread.p.  Please try to simplify it by getting rid of the classes and leaving just a simple "run" function that you will start using a basic Thread instance.  If you can reproduce the issue in fewer than 10 lines of code, I will take another look.
I will strongly recommend not using threads, but if you must, don't make async calls from the thread: use hndl instead of neg[hndl] in upd and use pubsub.listen() instead of pubsub.get_message().

Thanks for the advice. I've reduced the complexity of the code. Process 1 and Process 3 remain untouched. As for Process 2, I now have the following code, essentially a "dumb" listener:
Sync sending blocks the process too, so I've had to switch to async. I've also changed from get_message() to listen()
r. p Code
$ cat r.p
import sys
import redis
import threading
import cmd
import time

def callback():
    r = redis.StrictRedis(host='<IP_ADDRESS>', port=6379, db=0)

    channel = 'quote*'
    p = r.pubsub()
    p.psubscribe(channel)

    print('monitoring channel', channel)
    for m in p.listen():
        if m is None:
            continue
        else:
            q.upd(m)

def start_feed():
    my_thread = threading.Thread(target=callback)
    my_thread.setDaemon(True)
    my_thread.start()

q.start = start_feed

q)\l r.p
q)hndl:hopen 6006
q)upd:{[d] data:d; neg[hndl](`upd;`tbl; data)};
q)start()
q)('monitoring channel', 'quote*')

In parallel, I've spent the last 24 hours trying on different Linux builds (AWS r3.2xlarge running Ubuntu 16.04, AWS t2.micro running RHEL 7.4, AWS m4.xlarge running RHEL 7.4, personal desktop running Ubuntu 16.04), different Python versions (3.6.0, 3.5.2, 2.7.12, 2.7.5), q binaries (32-bit and 64-bit) and CPU affinity (taskset on one core or no tasksetting) the results are mixed:

Ubuntu 16.04 (desktop and AWS) always failing with 10-15 minutes

PyQ 4.1.3
KDB+ 3.5 (2018.02.26) l64
Python 2.7.12 (default, Dec  4 2017, 14:50:18)
[GCC 5.4.0 20160609

PyQ 4.1.3
KDB+ 3.5 (2018.02.26) l64
Python 2.7.12 (default, Dec  4 2017, 14:50:18)
[GCC 5.4.0 20160609]

t2.micro RHEL 7.4 still no errors --- 32-bit and 64-bit

PyQ 4.1.3
KDB+ 3.5 (2017.11.30) l32
Python 3.6.0 (default, May  9 2018, 08:11:26)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]

PyQ 4.1.3
KDB+ 3.5 (2018.02.26) l64
Python 2.7.5 (default, Feb 20 2018, 09:19:12)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]

m4.xlarge failed once, but since setting taskset to one core, it's been running smoothly

PyQ 4.1.3
KDB+ 3.5 (2018.02.26) l64
Python 2.7.5 (default, Feb 20 2018, 09:19:12)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]

FWIW, I have the 64-bit license which sends heartbeat packets to Kx's license daemon, but I doubt that could interfere...

I've simplified you r.p further as follows:
import redis
import threading
import time

from pyq import q

def run(r, channel):
    p = r.pubsub()
    p.psubscribe(channel)
    print('monitoring channel', channel)
    for m in p.listen():
        assert m is not None
        q.upd(m)

def start():
    t = threading.Thread(target=run,
        args=(redis.StrictRedis(host='<IP_ADDRESS>', port=6379, db=0),
              "quote*"))
    t.setDaemon(True)
    t.start()

q.start = start

q)hndl:hopen 6006;
  upd:{[d] data:d; neg[hndl](`upd;`tbl; data)};
  start();

This makes it self-contained and you can run it simply as q r.p.  I did get a crash when running under rlwrap:
q: warning: q crashed, killed by SIGSEGV.
q itself has not crashed, but for transparency,
it will now kill itself (without dumping core) with the same signal

warnings can be silenced by the --no-warnings (-n) option

but when I restarted without rlwrap, I don't see a crash anymore.
From your report, I see that you also use rlwrap.  Please try to reproduce the issue without rlwrap.

Thanks for getting back. I've just run as mentioned (without rlwrap). Again seg-fault... :(
(pyq2) zakariyya@zakariyya-PC-2193:~/Work/pyq2$ ./q/l64/q r.p
KDB+ 3.5 2018.02.26 Copyright (C) 1993-2018 Kx Systems
l64/ 4()core 15999MB zakariyya zakariyya-pc-2193 <IP_ADDRESS> EXPIRE 2018.07.01 zak********* KOD #51578

q)hndl:hopen 6006
q)upd:{[d] data:d; neg[hndl](`upd;`tbl; data)};
q)
q)start()
q)('monitoring channel', 'quote*')

q)Sorry, this application or an associated library has encountered a fatal error and will exit.
If known, please email the steps to reproduce this error to<EMAIL_ADDRESS>with a copy of the kdb+ startup banner.
Thank you.
./q/l64/q() [0x47a8b1]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7fa7bb84e390]
./q/l64/q(r0+0) [0x41bd40]
./q/l64/q() [0x408a9c]
./q/l64/q() [0x40fb6f]
./q/l64/q() [0x4042c8]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7fa7bb493830]
./q/l64/q() [0x4043a1]
Segmentation fault (core dumped)
(pyq2) zakariyya@zakariyya-PC-2193:~/Work/pyq2$ pyq --versions
PyQ 4.1.3
KDB+ 3.5 (2018.02.26) l64
Python 2.7.12 (default, Dec  4 2017, 14:50:18)
[GCC 5.4.0 20160609]

OK, after 300,000 messages, I did see a crash.  I have two ideas to try next:

Let's take redis out of the picture by replacing get_message() with dict(type='pmessage', ...).
I still blame the async IPC call from the thread.  Let's replace neg[hndl] with hndl in the listener's upd function.

I was successful in reproducing the problem without Redis.  The following script requires only a virgin kdb+ instance listening on port 6006:
$ cat r2.p
import threading
import time

from pyq import q

def run():
    while 1:
        q.upd()
        time.sleep(0.001)

def start():
    t = threading.Thread(target=run)
    t.setDaemon(True)
    t.start()

q.start = start

q)h:neg hopen 6006;
  upd:{h({n+::x};1)};
  start();

It reliably crashes after less than 10,000 cycles.  At this point we should probably report this to Kx, but I won't be surprised if they respond with the advise to move IPC calls to the main thread.

.. and finally, I have a proof that this issue has nothing to do with PyQ:
$ q -s 4
KDB+ 3.5 2018.04.25 Copyright (C) 1993-2018 Kx Systems
m32/ 20()core 65536MB **** NONEXPIRE

q)h:neg hopen 6006
q)upd:{h({n+::x};1)}
q)upd peach til 100000
[1]    74350 segmentation fault (core dumped)  q -s 4

(on the remote end, n got incremented to 4)
I am reassigning this issue to @awilson-kx to see what the Kx team has to say about this behavior.
For completeness, the back trace is
$ lldb -f $VIRTUAL_ENV/q/m32/q -c /cores/core.74350
(lldb) target create "/Users/a/.virtualenvs/3/q/m32/q" --core "/cores/core.74350"
Core file '/cores/core.74350' (i386) was loaded.
(lldb) process status
Process 0 stopped
* thread #1, stop reason = signal SIGSTOP
    frame #0: 0xa754745e libsystem_kernel.dylib`semaphore_wait_trap + 10
libsystem_kernel.dylib`semaphore_wait_trap:
->  0xa754745e <+10>: retl
    0xa754745f <+11>: nop

libsystem_kernel.dylib`semaphore_wait_signal_trap:
    0xa7547460 <+0>:  movl   $0xffffffdb, %eax         ; imm = 0xFFFFFFDB
    0xa7547465 <+5>:  calll  0xa7552f64                ; _sysenter_trap
  thread #2, stop reason = signal SIGSTOP
    frame #0: 0x0000af67 q`___lldb_unnamed_symbol61$$q + 951
q`___lldb_unnamed_symbol61$$q:
->  0xaf67 <+951>: movl   (%eax), %eax
    0xaf69 <+953>: testl  %eax, %eax
    0xaf6b <+955>: jne    0xaf1b                    ; <+875>
    0xaf6d <+957>: jmp    0xad0d                    ; <+349>
  thread #3, stop reason = signal SIGSTOP
    frame #0: 0x0002011a q`r0 + 10
q`r0:
->  0x2011a <+10>: movl   0x4(%eax), %edx
    0x2011d <+13>: testl  %edx, %edx
    0x2011f <+15>: je     0x20140                   ; <+48>
    0x20121 <+17>: movl   0x82ffe(%ebx), %ecx
  thread #4, stop reason = signal SIGSTOP
    frame #0: 0xa754745e libsystem_kernel.dylib`semaphore_wait_trap + 10
libsystem_kernel.dylib`semaphore_wait_trap:
->  0xa754745e <+10>: retl
    0xa754745f <+11>: nop

libsystem_kernel.dylib`semaphore_wait_signal_trap:
    0xa7547460 <+0>:  movl   $0xffffffdb, %eax         ; imm = 0xFFFFFFDB
    0xa7547465 <+5>:  calll  0xa7552f64                ; _sysenter_trap
  thread #5, stop reason = signal SIGSTOP
    frame #0: 0xa754745e libsystem_kernel.dylib`semaphore_wait_trap + 10
libsystem_kernel.dylib`semaphore_wait_trap:
->  0xa754745e <+10>: retl
    0xa754745f <+11>: nop

libsystem_kernel.dylib`semaphore_wait_signal_trap:
    0xa7547460 <+0>:  movl   $0xffffffdb, %eax         ; imm = 0xFFFFFFDB
    0xa7547465 <+5>:  calll  0xa7552f64                ; _sysenter_trap

Thanks a lot for taking the time to look into this, but also glad to know I was going crazy :sweat_smile:
I've actually already implemented the main thread solution, which funnily enough needs to make sync calls to the "Tickerplant", otherwise the packets get thrown God knows where.
I guess the split thread solution was an interesting approach to having q and Python run in parallel to manage their bits together (Python for Redis management, q for IPC and reconnecting to TP...)
Interestingly, the processes are still running on the AWS RHEL 7.4 boxes after now 10 hours (will leave them overnight but looks like they've passed the threshold of 10,000 cycles).


Bugfix: UnhandledPromiseRejectionWarning with singlefile attempt

Describe the bug
Decided to move forward with a clean archive, and keep the old one as a historical snapshot. Added a new list of links to archive, but after 6 or 7, almost all of the tasks start erroring out in cascade fashion, starting with singlefile step. Error is TimeoutExpired Command for chromium-browser, but it gives me a command to run, which also errors out with (node:162485) UnhandledPromiseRejectionWarning: SyntaxError: Unexpected number in JSON at position 2
Steps to reproduce

Installed with pip install archivebox
Changed config to disable MEDIA and timeout set to 180 seconds.
Added list of links with archivebox add ./list.txt.
Starts adding things correctly, but after link 5 or 6 starts to error out.

Screenshots or log output
Original error
[i] [2020-09-02 17:41:18] ArchiveBox v0.4.21: archivebox update
    > /mnt/volume/.archivebox-output/new-archive

[*] [2020-09-02 17:41:21] Writing 33 links to main index...
    √ /mnt/volume/.archivebox-output/new-archive/index.sqlite3
    √ /mnt/volume/.archivebox-output/new-archive/index.json
    √ /mnt/volume/.archivebox-output/new-archive/index.html

[▶] [2020-09-02 17:41:23] Collecting content for 8 Snapshots in archive...

[√] [2020-09-02 17:44:28] "NYPD used facial recognition to track down Black Lives Matter activist - The Verge"
    https://theverge.com/2020/8/18/21373316/nypd-facial-recognition-black-lives-matter-activist-derrick-ingram
    √ ./archive/1598996327.937664
      > singlefile
        Extractor failed:
            TimeoutExpired Command '['/mnt/volume/.archivebox-output/new-archive/node_modules/archivebox/node_modules/single-file/cli/single-file', '--browser-executable-path=chromium-browser', '--browser-args="["--headless", "--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36", "--window-size=1440,2000"]"', 'https://theverge.com/2020/8/18/21373316/nypd-facial-recognition-black-lives-matter-activist-derrick-ingram', '/mnt/volume/.archivebox-output/new-archive/archive/1598996327.937664/singlefile.html']' timed out after 180 seconds
        Run to see full output:
            cd /mnt/volume/.archivebox-output/new-archive/archive/1598996327.937664;
            /mnt/volume/.archivebox-output/new-archive/node_modules/archivebox/node_modules/single-file/cli/single-file --browser-executable-path=chromium-browser "--browser-args="["--headless", "--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36", "--window-size=1440,2000"]"" https://theverge.com/2020/8/18/21373316/nypd-facial-recognition-black-lives-matter-activist-derrick-ingram /mnt/volume/.archivebox-output/new-archive/archive/1598996327.937664/singlefile.html

      > pdf
        Extractor failed:
            Exception Failed to chmod: output.pdf does not exist (did the previous step fail?)
        Run to see full output:
            cd /mnt/volume/.archivebox-output/new-archive/archive/1598996327.937664;
            chromium-browser --headless "--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36" --window-size=1440,2000 --timeout=180000 --print-to-pdf https://theverge.com/2020/8/18/21373316/nypd-facial-recognition-black-lives-matter-activist-derrick-ingram

When I run
cd /mnt/volume/.archivebox-output/new-archive/archive/1598996327.937664;
/mnt/volume/.archivebox-output/new-archive/node_modules/archivebox/node_modules/single-file/cli/single-file --browser-executable-path=chromium-browser "--browser-args="["--headless", "--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36", "--window-size=1440,2000"]"" https://theverge.com/2020/8/18/21373316/nypd-facial-recognition-black-lives-matter-activist-derrick-ingram /mnt/volume/.archivebox-output/new-archive/archive/1598996327.937664/singlefile.html

Output:
(node:162485) UnhandledPromiseRejectionWarning: SyntaxError: Unexpected number in JSON at position 2
    at JSON.parse (<anonymous>)
    at getBrowserOptions (/mnt/volume/.archivebox-output/new-archive/node_modules/archivebox/node_modules/single-file/cli/back-ends/puppeteer.js:63:51)
    at Object.exports.initialize (/mnt/volume/.archivebox-output/new-archive/node_modules/archivebox/node_modules/single-file/cli/back-ends/puppeteer.js:36:35)
    at initialize (/mnt/volume/.archivebox-output/new-archive/node_modules/archivebox/node_modules/single-file/cli/single-file-cli-api.js:46:16)
    at run (/mnt/volume/.archivebox-output/new-archive/node_modules/archivebox/node_modules/single-file/cli/single-file:33:59)
    at Object.<anonymous> (/mnt/volume/.archivebox-output/new-archive/node_modules/archivebox/node_modules/single-file/cli/single-file:30:1)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
(node:162485) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:162485) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Software versions

OS:                        Ubuntu 20.04.1 LTS
ArchiveBox version:        0.21
Python version:            Python 3.7.9
Chrome version:            Chromium 84.0.4147.105

@drpfenderson there is a PR that should fix the issue with the index. You can give it a try (it should be merged this week tho): https://github.com/pirate/ArchiveBox/pull/452
About this issue, I will give it a check.

@cdvv7788 Thanks for pointing me that way.

Removed archivebox with pip uninstall archivebox
Made a new dir, cloned your repo into it, and checked out the sql_index branch.
Used pip install -e . to install.
It had some weird issues with the node_modules directory, so I removed it with rm -r and reinstalled with npm install --prefix . "git+https://github.com/cdvv/ArchiveBox.git#sql_index"
Ran archivebox init just in case.
Ran archivebox update.

Same errors in same places, including when I  Run to see full output:. The error message is identical, line numbers and all, to the error message listed under Output above. I tried removing the link as well with archivebox remove --delete https://theverge.com/2020/8/18/21373316/nypd-facial-recognition-black-lives-matter-activist-derrick-ingram and readding it, but get the same timeout error.
Oh well. Guess I should wait until the next release.

@drpfenderson It works for me. Can you try running it with docker or installing the npm dependencies? (the quickstart has been updated with the instructions)

@cdvv7788 I get the same timeout using docker/docker-compose, though I am really not sure how to run the node_modules commands from inside the docker to see the exact error. If I cd to the directory listed, don't I lose the docker-compose.yml and the path for node_modules? I tried researching the exec/run commands a bit to figure out how to execute the single-file command from within a container, but I can't really grok it.
$ docker-compose run archivebox add https://theverge.com/2020/8/18/21373316/nypd-facial-recognition-black-lives-matter-activist-derrick-ingram

[i] [2020-09-03 15:55:38] ArchiveBox v0.4.21: archivebox add https://theverge.com/2020/8/18/21373316/nypd-facial-recognition-black-lives-matter-activist-derrick-ingram
    > /data

[+] [2020-09-03 15:55:41] Adding 1 links to index (crawl depth=0)...
    > Saved verbatim input to sources/1599148541-import.txt
    > Parsed 1 URLs from input (Plain Text)
    > Found 1 new URLs not already in index

[*] [2020-09-03 15:55:41] Writing 33 links to main index...
    √ /data/index.sqlite3
    √ /data/index.json
    √ /data/index.html

[▶] [2020-09-03 15:55:42] Collecting content for 1 Snapshots in archive...

[+] [2020-09-03 15:55:42] "theverge.com/2020/8/18/21373316/nypd-facial-recognition-black-lives-matter-activist-derrick-ingram"
    https://theverge.com/2020/8/18/21373316/nypd-facial-recognition-black-lives-matter-activist-derrick-ingram
    > ./archive/1599148541.818964
      > title
      > favicon
      > wget
      > singlefile
        Extractor failed:
            TimeoutExpired Command '['/node/node_modules/single-file/cli/single-file', '--browser-executable-path=chromium', '--browser-args="["--headless", "--no-sandbox", "--disable-gpu", "--disable-dev-shm-usage", "--disable-software-rasterizer", "--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36", "--window-size=1440,2000"]"', 'https://theverge.com/2020/8/18/21373316/nypd-facial-recognition-black-lives-matter-activist-derrick-ingram', '/data/archive/1599148541.818964/singlefile.html']' timed out after 180 seconds
        Run to see full output:
            cd /data/archive/1599148541.818964;
            /node/node_modules/single-file/cli/single-file --browser-executable-path=chromium "--browser-args="["--headless", "--no-sandbox", "--disable-gpu", "--disable-dev-shm-usage", "--disable-software-rasterizer", "--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36", "--window-size=1440,2000"]"" https://theverge.com/2020/8/18/21373316/nypd-facial-recognition-black-lives-matter-activist-derrick-ingram /data/archive/1599148541.818964/singlefile.html

Author of SingleFile here. This is a a bug in SingleFile due to the presence of "," in the --browser-args switch. I'm trying to see how it could be fixed or circumvented. I guess proposing to pass a JSON string was not a good idea.

Finally, I was able to fix the issue by formatting the browser-args like this:
--browser-args=\"\"[\"\"--headless\"\", \"\"--no-sandbox\"\", \"\"--disable-gpu\"\", \"\"--disable-dev-shm-usage\"\", \"\"--disable-software-rasterizer\"\", \"\"--window-size=1440,2000\"\", \"\"--user-agent\"=\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36\"\"]\"\"

Thanks. We definitely need to improve the command so it can be run directly on case of error. I will add that to the pending tasks.
@drpfenderson What is your TIMEOUT setting? Have you modified any setting or changed any environment variable? I cannot reproduce it, not even with docker, so I am suspecting you have something unusual in your local setup.

@drpfenderson I just created a PR that should escape correctly the argument. With that you should be able to, at least, run the command properly, and come back with the error that singlefile is returning.

Well, I essentially did a find/replace/delete for every archivebox/node/python file on my system that could be related. There were a number of weird places with files due to the various installation methods I've used over the years for this program. Realizing the amount of work to disentangle everything, I spun up a new server, attached the archive to it, wiped all the local config/conf files, ran the docker-compose run archivebox add and it all worked!
Thank you all for your infinite patience with me.


Compatibility with upstream Httpaf (Opium)

Hi there! I am currently working on a web server that serves an API which is itself a combination of multiple other APIs wrapped around a standardized interface. As such, my project needs both an http server library (to handle incoming requests) and an http client library (to forward those requests to the actual target APIs). I am using Opium (which depends on upstream Httpaf) as my main server library and I am trying to use Piaf (which depends on your fork of Httpaf) as my http client library. However, I am having the hardest time getting the project to compile because of conflicts between the two versions of Httpaf.
Any suggestions on how I can get this to work?
Here is the build error in question:
File "opium/src/app.ml", line 29, characters 31-32:
29 |     Rock.Server_connection.run f app
                                    ^
Error: This expression has type
         request_handler:(Httpaf.Reqd.t Gluten.reqd -> unit) ->
         error_handler:Httpaf.Server_connection.error_handler -> unit Lwt.t
       but an expression was expected of type
         request_handler:Httpaf.Server_connection.request_handler ->
         error_handler:Httpaf.Server_connection.error_handler -> 'a Lwt.t
       Type Httpaf.Reqd.t Gluten.reqd -> unit is not compatible with type
         Httpaf.Server_connection.request_handler = Httpaf.Reqd.t -> unit
       Type Httpaf.Reqd.t Gluten.reqd = Httpaf.Reqd.t Gluten.Reqd.t
       is not compatible with type Httpaf.Reqd.t

Here is part of my package.json (I am using esy):
"dependencies": {
  ...
  "@opam/httpaf-lwt-unix": "*",
  "@opam/httpaf-lwt": "*",
  "@opam/httpaf": "*",
  "@opam/piaf": "*",
  "@opam/opium": "~0.20.0"
},
"resolutions": {
  "@opam/httpaf-lwt-unix": "anmonteiro/httpaf:httpaf-lwt-unix.opam#0555dde898f0886fb9909005e00b7527cc1a3895",
  "@opam/httpaf-lwt": "anmonteiro/httpaf:httpaf-lwt.opam#0555dde898f0886fb9909005e00b7527cc1a3895",
  "@opam/httpaf": "anmonteiro/httpaf:httpaf.opam#0555dde898f0886fb9909005e00b7527cc1a3895"
}

Unfortunately this library is not compatible with upstream http/af, as it uses my forked version of it.
This specific interface incompatibility can be attributed to me adding support for upgrading connections (to websockets, for example).
That also means Opium can't work with it in its current state.

My resolutions
{
  "resolutions": {
    "@opam/httpaf": "github:anmonteiro/httpaf:httpaf.opam#559019829b7ed267a5b4b86aed1e7d7795214bcd",
    "@opam/httpaf-lwt": "github:anmonteiro/httpaf:httpaf-lwt.opam#559019829b7ed267a5b4b86aed1e7d7795214bcd",
    "@opam/httpaf-lwt-unix": "github:anmonteiro/httpaf:httpaf-lwt-unix.opam#559019829b7ed267a5b4b86aed1e7d7795214bcd",
    "@opam/piaf": "github:EduardoRFS/piaf:piaf.opam#b93282e9c60d193d5c3572009d71b2cf3e4b81ca",
    "@opam/ssl": "github:Firgeis/ocaml-ssl:ssl.opam#f7524f7b6db756ce4097dea7fa4e1c4f4f2832a9",
    "@opam/opium": "github:EduardoRFS/opium:opium.opam#9fdbfbd0eedf238e1be0663ab95977be27d9a6e1",
    "@opam/rock": "github:EduardoRFS/opium:rock.opam#9fdbfbd0eedf238e1be0663ab95977be27d9a6e1"
  }

no longer relevant.


Change tags of multiple items at once

Hi iD-Team!
The feature I'm missing most in iD is to change tags of multiple selected items at once.
For example: I want to just add name & ref to a 10 Km-road, which consists of 100 consecutive 100 meter ways. Right now i've to select each way, and add name+ref 100 times! :-(
Much better would be to multi select (by holding down "Shift" or "Ctrl"-key) and left click lets say 30 or even each of the 100 ways and then ONCE assign name+ref to all of them.
That would improve and speed up tagging of long roads dramatically.

Thanks for the suggestion. See #1761 where this has previously been requested.


Separate core test and SDK usage test

Is your feature request related to a problem? Please describe.
SDK tests (with the examples) depend on secret keys, it means that on pull_request workflows wont' work. We want to be able to test PRs. The tests are currently running on pull_request_target and on main. It might make sense for testing core.
Describe the solution you'd like
Separate SDK tests from bolt core program tests (world, bolt-component, bolt-system).
SDK examples keypairs could be dynamically generated.

In fact, would it even possible to not rely on any secret for testing the code? :)


Replaced deprecated node-sass with sass

Since Bootstrap 5, recommended is to use sass library, as node-sass is deprecated, even if we are still using Bootstrap 3 in the demo, this change should be considered.
@javiereguiluz (Sorry for the ping 🙏) Should I also resolve #1342 here?

As you said, we're using a legacy version of Bootstrap (there's a pending PR to update it ... but it stalled and I don't know if it's mergeable). Do you know if updating node-sass to sass is safe for such a legacy version? Thanks!
Yes, if you prefer, we can do all the updates in a single PR. Cheers!

The only concern I would see here is the deprecation of Slash as Division, which is still used in Bootswatch, but I don't think this deprecation will be removed anytime soon. (We will probably upgrade the demo to a newer version of Bootstrap sooner.)

Thank you @skmedix.


Move base-entity-* to store package

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@3058d8a). Click here to learn what that means.
The diff coverage is 97.43%.

@@            Coverage Diff            @@
##             master    #4371   +/-   ##
=========================================
  Coverage          ?   58.52%
=========================================
  Files             ?      921
  Lines             ?    30011
  Branches          ?     4352
=========================================
  Hits              ?    17563
  Misses            ?    12448
  Partials          ?        0


Element Types JSX.IntrinsicElements vs React.[Element]HTMLAttributes*

Thank you for all the work you have done on this library, it's helped tremendously.
Not a bug report but just a question around typings in this library.
I am creating my own select/button components etc and usually use React.[Element]HTMLAttributes<HTML[Element]Element types instead of using JSX.InstrinsicElements["element"], which this library uses. Main issue is that JSX.IntrinsicElements always passes a ref as a prop which has the LegacyRef type rather than Ref, and always passes a ref even when it will not actually work.
Example, my button component is as below (removed imports and variants for brevity):
const CustomButton = React.forwardRef<HTMLButtonElement, ButtonProps>(
  ({ className, variant, size, ...props }, ref) => {
    return (
      <button
        className={clsx(buttonVariants({ variant, size, className }))}
        ref={ref}
        {...props}
      />
    )
  }
)

buttonComponent import from the above:
const Button = (props: JSX.IntrinsicElements['button']) => (
  <CustomButton
    className="w-full justify-center"
    variant="default"
    {...props}
  />
)

In this scenario, the incompatible LegacyRef is forwarded through as a prop from JSX.IntrinsicElements
TS Feedback:
Type 'LegacyRef<HTMLButtonElement> | undefined' is not assignable to type 'Ref<HTMLButtonElement> | undefined'.
      Type 'string' is not assignable to type 'Ref<HTMLButtonElement> | undefined'.

Forgive my naivety if this is a specific design decision, this article goes a little more in depth into the issues I am running into.

Hey, @micotodev! Thanks for the issue and the article. I wasn't aware of this difference. I believe we can be more explicit about our types with or without ref in v2.
We also have #121, where we'll try to make the whole component props generic in v2. But I'll leave this issue open in case we can't make them generic enough and want to be more explicit.
Thank you again!


don't try to spilt None - fixes AttributeError: 'NoneType' object has no...

I broke the special commands :( sorry!!

No worries. Thanks for noticing it and fixing it. :)
Cheers!
:train:


After calling the unload method, black screen is shown on android

Hello!
Thanks for maintaining this library!
I have noticed this kind of behavior on android:
Open example app > Open screen with unity scene > Unload & Go back > Open screen with unity scene again > Unity scene is black.
Video demo:
https://user-images.githubusercontent.com/46750002/208429244-e068ace0-709c-46d4-b617-bda3a33faadd.mp4
Relevant files:

UnityScreen.txt

MacOS: 13.01
The example project is taken from @azesmway's zip file, exported from unity by him, in other issues' comment sections in this repository but this issue happens of project in this repository too.
Does anyone have any idea if the fix for this issue exists?

Same issue, but no need to unloadUnity, just destroying and recreating the component simply lead to blank screen (white screen in my case).

+1

I added a loading indicator and wait for 1 second each time I come to my unityView screen.
{loading ? (
        <Loader color={colors.primary} />
      ) : (
        <UnityView
          ref={unityRef}
          style={{flex: 1}}
          onUnityMessage={result => {
            console.log('onUnityMessage:', result.nativeEvent.message);
          }}
        />
      )}

added useEffect focus function of navigation and in that function I made loading true and after one second I made it false and then I can see the unityView instead of black/white screen.

Sounds interesting. Would you mind sharing more details about the "useEffect focus function"? Did you mean useFocusEffect? I tried the following, but it resulted in segment fault:
const [ loading, setLoading ] = useState(false)
useFocusEffect(() => {
  setLoading(true)
  setTimeout(() => setLoading(false), 1000)
})
if (loading) {
  return (
    <Text>Loading...</Text>
  )
}
return (
  <UnityView
    style={{ flex: 1 }}
  />
)

After looking into the source code, I found androidKeepPlayerMounted is another workaround. It keeps the player mounted so that we can manually pauseUnity on blur and resumeUnity on focus. It does not lead to any null pointer dereference or segment fault because the lifecycle is simpler in this case.

After looking into the source code, I found androidKeepPlayerMounted is another workaround. It keeps the player mounted so that we can manually pauseUnity on blur and resumeUnity on focus. It does not lead to any null pointer dereference or segment fault because the lifecycle is simpler in this case.

Thanks for finding this! It was very helpful. I ended up using the androidKeepPlayerMounted option and useFocusEffect with pauseUnity on blur and resumeUnity on focus and it worked like a charm.
Considering the naming of the option, any Idea if this would work on IOS? I plan on testing it but figured I'd ask first to determine if there was another method for IOS or whether it'd be tested yet.

It seems on iOS the Unity instance is kept alive in the background already (not sure if it is "mounted" or not).

@Luluno01 Thank you. That's good to know concerning iOS.
Although the Android mounting method works great, my use case necessitates that I completely restart my game when a new user logs in. As such, I tried using the unload method along with the delay that @aawolf1099 mentions, but it leaves the game in an undesirable state and trying to rerender the game again crashes the app.

@Luluno01 Thank you. That's good to know concerning iOS.
Although the Android mounting method works great, my use case necessitates that I completely restart my game when a new user logs in. As such, I tried using the unload method along with the delay that @aawolf1099 mentions, but it leaves the game in an undesirable state and trying to rerender the game again crashes the app.

This is perfectly legit requirement. Unfortunately, it seems Unity doesn't implement any interface to restart itself on Android. My current approach is to unload everything from within the game and reload the scene, which helps a bit, but if you are using some Unity plugins or features that touch native code or native resources, it might be insufficient to just reload the scene.
Considering how unstable it is to embed Unity view into React Native, I start to think if not embedding it directly is a better idea. On Android, it is easy to do so. You just add android:process=":unity" to UnityPlayerActivity and use an intent to start a separate Unity activity (communication between React Native and Unity will be another problem). It is then easy to restart the Unity process without killing the entire app. On iOS, however, I haven't found a similar solution where you can start a new task window that runs in a separate process but still belongs to the same app.

Actually I just found another workaround. I found that when there is a black screen, if I send the app to the background and then switch back, it starts to render properly. So, I made some changes (#78) to the binding to expose UnityPlayer.windowFocusChanged, which is invoked under the hood when the app is sending to the background and bringing to the foreground, and it does get the view out of black screen.


Describe object vs Object in the Handbook

https://www.typescriptlang.org/docs/handbook/basic-types.html should cover the built-in object type and have a brief discussion of it vs Object
See #20614

@RyanCavanaugh May I help with this? If I can would you like me to create a new section for object vs Object or do you have something else in mind?


[WIP] Dynamic Metadata

This is a work in progress solution for #555 which provides the ability to update metadata on a running instance.  Please take a look at the progress so far to see if the approach makes sense.

Metadata has been moved out of the /object key in etcd and into individual etcd keys under /metadata to allow for individual updates to metadata values without fear of stepping on each other.
A new endpoint PATCH /machines/<mach_id> was created to handle metadata updates for a single machine. It takes a json document with 1 or more keys to update, example below.
Metadata supports per-value TTL.
Metadata keys can be removed by setting their value to null or empty string
Metadata in the machine heartbeat is ignored unless it is the first time it is seen. If the machine state needs to be created in etcd, the metadata in the heartbeat will be used as the starting point.

Example PATCH request
{
   "foo": { "value": "bar" },
  "baz": {"value": "stuff"; "ttl": 30},
  "feh": {"value": null }
}

I think this is ready for the first pass of reviews.

I haven't looked too far into the actual code, but I wanted to share some high-level feedback since this is likely to change some things around:
I'd like to see the PATCH endpoint implemented using JSON patch [0] rather than a custom request format. Additionally, we should consider implementing the PATCH at the root of the machines collection rather than per-machine.
What utility does the TTL on a specific key provide?
[0] http://jsonpatch.com/

Switching to JSONPatch is easy enough as is moving the PATCH at the root level to allow multiple machines to be updated at once. I'll have the changes made in a couple of days.
The original idea was to use it as kind of a health check, a process could check a node to see if it was suitable for a specific type of job and set the TTL'ed metadata. Furthur thoughts make it seem like the TTL idea is half baked / gold plating and maybe doesn't deserve to live, particularly with the new request format.

@epipho I do like the idea, but I'm not sure if we should try to squeeze it in right now.

@bcwaldon

Updated to use jsonpatch on the machines collection level.  Metadata is now updated with the path format /machineid/metadata/key using the add/remove/replace ops.

[{"op": "add", "path": "/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/metadata/foo", "value": "bar"},
	 {"op": "replace", "path": "/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb/metadata/ping", "value": "splat"}]

TTL feature has been stashed away for another day

Error handling: Currently the handler does as much as it can to validate the patch data before doing any work. Invalid ops, bad path format, or missing values will cause the request to fail with no effect on the data.  If an error occurs during the apply phase, the request terminates immediately and returns the error. Because the changes cannot be made atomic as the metadata keys are split apart this could (rarely) result in a partial patch being applied. Any suggestions on how to handle this or what information should be returned to the caller when this happens?

Removing WIP. Ready for initial review and comments.

How do I test the dynamic metadata?
I am looking for a safe way to change the CoreOs machine metadata in run-time
Is this feature is the only way? (I blocked CoreOs updates on my cluster..)

@epipho In addition to the inline comments, I have a concern about how the metadata is initially making it into etcd. Machines leave and rejoin the cluster frequently, so metadata that was manually added by someone through etcd will get blown away. I wonder if the metadata published by a given machine should be managed independently from the metadata added through etcd? I think it makes sense for metadata manually added through etcd directly should persist until it is manually removed.

@bcwaldon Thank you for taking the time to do a very thorough review. I will address all the small stuff this weekend.
As for your question above:
Wouldn't that be true today since a machine leaving the cluster removes the /machineid/object value that currently holds the metadata?
I'm not sure how to best handle your request, I outlined a few possibilities below.

Metadata that is configured via fleet config (file, env variable, cloud config) is automatic, removed by machine leave, and cannot be manipulated at all. All other metdata values are untouched.

Metadata configured by fleet config can be manipulated, but will be reset by the machine rejoining. All metadata values are untouched when a machine leaves.

Metadata configured by fleet config is only added if the machine does not exist in the registry at all. All metadata values are untouched when a machine leaves.

Same as 3, but all metadata is removed when a machine leaves (as currently written).

I worry that separating the metadata will be confusing as some will be able to be manipulated via the api and some will not. Of all the suggestions outlined I like 3, treating the configured values as defaults if the machine has never been seen before but leaving it untouched otherwise.

@epipho It's not clear to me how it should work, so I'd like to get an opinion from @jonboulle on this.

All feedback has been addressed, just waiting for clarification on the metadata persistence issue.

@bcwaldon  @jonboulle  any thoughts yet on what to do about persistent metadata?

I could have sworn I wrote a reply to this ages ago but apparently not? :-(
I am still going back and forth but keep coming back to the thought that the sane behaviour would be to have user-set metadata stored completely separately from fleetd-set metadata, and then just make a decision on a simple merge strategy (likely, user-set-always-wins).

Seems reasonable. How does this feel:

Fleet metadata returns to the object.
User metadata is stored separately as above.
User metadata is never removed from etcd
When loaded, any values in user metadata overwrite those from fleet
metadata.
If a user issues a delete against a key in the fleet metadata, an empty
value is added to the user metadata to ensure the key remains removed if
the machine leaves and rejoins.

This allows for sane user-facing behavior as well as preserving the
behavior of changing the metadata in the fleet config and restarting the
service.
On Wed, Feb 11, 2015 at 6:26 PM, Jonathan Boulle<EMAIL_ADDRESS>wrote:

I could have sworn I wrote a reply to this ages ago but apparently not?
:-(
I am still going back and forth but keep coming back to the thought that
the sane behaviour would be to have user-set metadata stored completely
separately from fleetd-set metadata, and then just make a decision on a
simple merge strategy (likely, user-set-always-wins).
—
Reply to this email directly or view it on GitHub
https://github.com/coreos/fleet/pull/1077#issuecomment-73987582.

Sounds reasonable to me, @bcwaldon ?

@epipho What does it mean for a user to issue a delete against a key in the fleet metadata?

@bcwaldon I should have called them out as "user metadata" and "machine metadata".
If a user deletes a key defined by machine metadata (fleet config file, env variables in fleet.service) an entry will be added with an empty string to the dynamic user metadata to ensure the value stays deleted even if the machine leaves and rejoins the cluster.

Merging machine and user metadata has been implemented.

@epipho can you update the PR description with the current state of things? We need some documentation written up for this new feature, too.

If a unit is scheduled to a machine based on metadata, and that metadata changes so the the unit no longer matches, should it be rescheduled elsewhere?

@pwhiteside yes, if the machine no longer satisfies the unit's requested metadata the reconciler will reschedule it.

@bcwaldon I have updated the first post with all the current information. I am currently investigating the best way to make global units work properly with dynamic metadata, after which I will work on docs.

Global units now obey dynamic metadata, see 5941e1f.  This felt like the least intrusive way to ensure that agents had the full machine state during reconciliation.
I did find a strange quirk.
When the metadata is updated such that an instance of a global unit needs to be stopped fleetctl list-units displays instances of the unit incorrectly for ~10 seconds
The only relevant log appears to be
│Mar 01 01:41:07 core-02 fleetd[7113]: INFO manager.go:143: Triggered systemd unit test.service stop: job=24045
01 01:41:07 core-02 fleetd[7113]: INFO manager.go:273: Removing systemd unit test.service
01:41:07 core-02 fleetd[7113]: INFO reconcile.go:315: AgentReconciler completed task: type=UnloadUnit job=test.
│service reason="unit loaded but not scheduled here"

The units running on the other nodes appear to disappear and/or flap.  This is an output from fleetctl list-units every 2 seconds.  test.service is a global unit.
UNIT            MACHINE                         ACTIVE  SUB
test.service    1d2dca9f.../<IP_ADDRESS>        active  running
test.service    5f85299b.../<IP_ADDRESS>        active  running
test.service    c39e8d31.../<IP_ADDRESS>        active  running
UNIT            MACHINE                         ACTIVE  SUB
test.service    1d2dca9f.../<IP_ADDRESS>        active  running
test.service    5f85299b.../<IP_ADDRESS>        active  running
test.service    c39e8d31.../<IP_ADDRESS>        active  running

Change the metadata so that 1d2dca9f is no longer eligible to run the global unit.
UNIT            MACHINE                         ACTIVE  SUB
test.service    1d2dca9f.../<IP_ADDRESS>        active  running
test.service    5f85299b.../<IP_ADDRESS>        active  running
test.service    c39e8d31.../<IP_ADDRESS>        active  running
UNIT    MACHINE ACTIVE  SUB
UNIT    MACHINE ACTIVE  SUB
UNIT    MACHINE ACTIVE  SUB
UNIT            MACHINE                         ACTIVE  SUB
test.service    5f85299b.../<IP_ADDRESS>        active  running
UNIT            MACHINE                         ACTIVE  SUB
test.service    5f85299b.../<IP_ADDRESS>        active  running
UNIT            MACHINE                         ACTIVE  SUB
test.service    5f85299b.../<IP_ADDRESS>        active  running
UNIT            MACHINE                         ACTIVE  SUB
test.service    5f85299b.../<IP_ADDRESS>        active  running
test.service    c39e8d31.../<IP_ADDRESS>        active  running
UNIT    MACHINE ACTIVE  SUB
UNIT    MACHINE ACTIVE  SUB
UNIT    MACHINE ACTIVE  SUB
UNIT    MACHINE ACTIVE  SUB
UNIT            MACHINE                         ACTIVE  SUB
test.service    5f85299b.../<IP_ADDRESS>        active  running
UNIT            MACHINE                         ACTIVE  SUB
test.service    5f85299b.../<IP_ADDRESS>        active  running
UNIT            MACHINE                         ACTIVE  SUB
test.service    5f85299b.../<IP_ADDRESS>        active  running
test.service    c39e8d31.../<IP_ADDRESS>        active  running
UNIT            MACHINE                         ACTIVE  SUB
test.service    5f85299b.../<IP_ADDRESS>        active  running
test.service    c39e8d31.../<IP_ADDRESS>        active  running
UNIT            MACHINE                         ACTIVE  SUB
test.service    5f85299b.../<IP_ADDRESS>        active  running
test.service    c39e8d31.../<IP_ADDRESS>        active  running

GET /v1/state returns the same information during this period.
The actual units on the other machines do not actually exit or restart.  The same thing happens in master if you change the unit file and restart the service so I don't think it is related to there changes.

@bcwaldon  Got stomped by work, I will be back to this this weekend.
I believe the only outstanding item at this point is docs.

Added a few bits of documentation about the PATCH machines api. Not sure where to put the explanation on how metadata merging works. Architecture doc perhaps?

@bcwaldon  What do we still need here?

+1 for this

+1 for this

+1

+1; this would solve so many problems for us.

+1 !

@epipho @bcwaldon @jonboulle What's the status on this?

bump

:+1:

It has been a while since this issue has seen some activity.
This feature would be of great use!

Hi @epipho, I've been running your code for a month now and it works just fine. Thanks for your contribution!
I was wondering where I could find the code where you implemented the support for TTL?

There's one point I was wondering about lately: the inability to set a property with an empty value.
For example, sending a PATCH request as:
[ { "op": "add", "path": "/e7afc665c2dc4a4b82082cd654fc6555/metadata/test", "value": "" } ]
Returns:
{"error":{"code":400,"message":"invalid value: add and replace require a value"}}
Aren't empty properties valid if configured via /etc/fleet/fleet.conf?

Just a short note:
My thought about this PR is simply, "sure, why not?".
It sounds good in general. With code reviews and writing additional tests, we might merge it. I don't think anyone is against this one.
But when I tried to rebase it on top of current master, I somehow gave up. It's a little messy. Especially the newline characters at the end of every line in the document... OMG.
If someone could rebase it for me, I would really appreciate, be happy to review the code, and test it too.

I can definitely rebase the code for a start.

Well, easier said than done, not being a GitHub expert...
I have imported this PR's code into a branch in my own repo, and made it up to date with master: https://github.com/dalbani/fleet/tree/dynamic-metadata
Am I supposed to submit a new pull request then?

@dalbani Yes, please create a new PR. That would be appropriate, as the original author doesn't seem to be active any more. I'll try to review & test that next week.
Actually what I wanted was rebasing this branch on top of master, not merging master into the branch. Anyway let's discuss about that, looking into the new PR.

I have rebased on top of master, instead of merging, and created the new PR #1642.

Closed in favor of https://github.com/coreos/fleet/pull/1642. Thanks.


Makefile: $(BINARIES) depends on "vendor"

Fixes issue

Describe the change
Properly state the dependence between the binaries targets and vendor.
It prevent make parallel builds of the  "binaries" target (i.e. make -j binaries) to fail with errors ... is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
Re https://github.com/Homebrew/homebrew-core/pull/153571
How to verify it
make -j binaries
Changelog text
Enable parallel builds of the make "binaries" target.
Please verify and check that the pull request fulfills the following requirements

[ ] Tests have been added or not applicable
[ ] Documentation has been added, updated, or not applicable
[x] Changes have been rebased to main
[x] Multiple commits to the same code have been squashed

LGTM, thanks!

to be honest, I dont think we even need vendor build at all. so I went ahead filing a separate PR on it, https://github.com/regclient/regclient/pull/589


App install

Hi!
I'm not able to install the application.
$ make rel
/usr/bin/env: escript: No such file or directory
make: *** [rel] Error 127
Coul you give my some advice?
Regards.

It was necessary to install erlang for the /usr/bin/escript file.
Now I have another issue:
=ERROR REPORT==== 27-Oct-2016::11:25:34 ===
beam/beam_load.c(1682): Error loading module bbmustache:
This BEAM file was compiled for a later version of the run-time system than R16B03-1.
To fix this, please recompile this module with an R16B03-1 compiler.
(Use of opcode 156; this emulator supports only up to 153.)
===> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace
=ERROR REPORT==== 27-Oct-2016::11:25:34 ===
Loading of /home/centos/observerweb/rebar3/bbmustache/ebin/bbmustache.beam failed: badfile

Was an Erlang version isssue.
Now 19.1-1 installed and the app is running


Spacy v3 support

Just want to re-ask about #1621
I have noticed that snorkel still doesn't support spacy v3. I can help with this, but I think the info probably needs to update.
So right now for snorkel, which files use spacy?

Hi @yinxiangshi, the same set of files use spacy as before: https://github.com/snorkel-team/snorkel/issues/1621#issuecomment-779496657

Oh, thank you.

@henryre is there any way to stop certain tickets from getting cleaned every so often? A valid definition of a feature is open source product management and your obfuscating even the ones I’m sure you want.

@rjurney Hello, I've added preliminary spacy v3 support in pr. Could you please help review the changes? I am not that sure wrapper still needs changes. Thanks.

@yinxiangshi yes, it so happens I just left my job so I have time and need something to do :) Let me just set up my computer and I’ll get on it!

@rjurney In my perspective, I think we just need to add new parameters of _nlp in the preprocessor. I didn't see any other needed changes I think. Plz help me! Thanks!

@yinxiangshi Got my new computer setup, looking now!

@rjurney This is what I am looking forward :) Cong about your new life & new computer!

@yinxiangshi the second booster wiped me out like moments after I posted that... here goes again!

@yinxiangshi I guess we are using #1621 now?

Sorry for the dupe comment, but:
@yinxiangshi I got tox -e complex to run. I am looking over the relevant files to see if there are anything we missed. I didn't quite get your comments about config - I am not sure how that changes things, unless we want to add spaCy config support to snorkel. I suppose that is reasonable, let me look!


Remove preprocessor directives in function-like macros.

This PR fixes issue #5366 by reworking the preprocessor directives to surround the function-like macro rather than occurring inside the function-like macro parameter list.
This change allows the GDMP project to successfully build its windows target rather than failing with:
.\mediapipe/tasks/cc/core/task_api_factory.h(86): error C2121: '#': invalid character: possibly the result of a macro expansion

@kuaashish
Thank you for making the necessary changes to make this PR possible.
Please help us to get this PR approved.
https://github.com/google/mediapipe/issues/5366#issuecomment-2099717657

@Malcolmnixon
Gentle reminder.  Do we need to keep this issue open?
https://github.com/google/mediapipe/issues/5366

Do we need to keep this issue open?

I would assume so - although that may be up to the ticketing policies of this project. I'm used to policies where an issue remains open until such time as a solution has been "merged" which fixes it. This PR is one possible solution; however the core team may request additional changes, or a different PR may get merged before this one which also fixes the issue - thus rendering this PR unnecessary.

Thx for feedback
I will remind right actions to happen

Hi @schmidt-sebastian,
Could you please review this PR?
Thank you!!

@kuaashish
@schmidt-sebastian
Do seriously consider no more experimental Windows build
So the Unity and Godot communities will have the latest CI windows build without having someone knowing how to debug and ensure successful windows build
[Mediapipe supports Windows] No more experimental build

Do seriously consider no more experimental Windows build

While this PR does allow the code to compile under Visual Studio, it's more about having the mediapipe code avoid use-cases undefined by the C/C++ specification, and thus being more compatible with different compilers.
A deeper discussion about official Windows platform support should probably occur on a separate issue, as there may be many other contributing factors (such as 3rd party dependencies, or spaces in filenames disrupting scripts) resulting in the mediapipe team deciding Windows support warrants being kept as experimental.

@Malcolmnixon
Thanks for investing lots of time in getting CI windows build using VS2019
There are many c++ projects attempting to do that. My overview impression is that NONE succeed.
They remain in e.g. v0.10.9, due to one or two heros who managed to implement significant modifications to get it compile in e.g. VS20219
The Unity and TLife repos remain unsuccessful in implementing a CI to access the latest version.
I hope google team can address that so many projects can benefit IMMEDIATELY of the latest AI innovation coming from google.

@schmidt-sebastian
Can you review?


A tentative file format

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.2 out of 4 committers have signed the CLA.:white_check_mark: fleapapa:white_check_mark: dwayneberry:x: VrajPandya:x: asuhanYou have signed the CLA already but the status is still pending? Let us recheck it.


kube-dns addon fails in air gapped environment

the mandatory-addons-0-kube-dns task fails in an air gapped environment due to inability to pull the following resources
    "container": {
      "docker": {
        "kubedns": "gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.5",
        "kubedns-dnsmasq": "gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.5",
        "kubedns-sidecar": "gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.5",
        "heapster": "gcr.io/google_containers/heapster-amd64:v1.4.2",
        "dashboard": "gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.3"
      }
    },

Hi, linben
I also have the same issue while met installing kubernetes on DC/OS(community edition)
following is my last error message that i got.
Deploying kube-dns components --
configmap "kube-dns" configured
service "kube-dns" configured
deployment "kube-dns" configured
kube-dns check failed and is not ready
Rest of other containters come up well except mandatory-addon-0-kube-dns.
Do you still have the issues? or already resolved?
If you have resolved then, share your solution.
Thanks,
Ted

Hi @ted-jung
In addition to the above images, you need gcr.io/google_containers/pause-amd64:3.0
Make those images available locally in your image registry. Then run kubectl replace -f <component.yaml> for kube-dns, heapster, and dashboard (after modifying the configs to point to your private registry images).
Do kubectl get pods --all-namespaces to see status of these components
The next release will incorporate these changes.
-ben

@ted-jung @linben In our next release, we'll include a fix to tackle this issue. In any case, the registry from which to pull these docker images can be configured in our package via resource.json although we recommend to use the gcr.io/google_containers registry to avoid undesired issues.
In the following releases, we might include some flags to disable the deployment of heapster and dashboard.

Should be fixed in 0.3.0-1.7.10-beta.


Art to screen pipeline

Make a script (?) which uses png2src to get art into code
Draw those things in zig
Extend to work with a sprite sheet for animation

#76 should finally put this to rest.


Usage of remark-rehype options

This documents how to use the changes made in https://github.com/withastro/astro/pull/3297 in order to close https://github.com/withastro/astro/issues/3163

Hi @gtnbssn ! I'm just going through the older PRs still marked as drafts, and I can see that the original PR for the changes to astro core repository hasn't seen any activity in a while.
Just pinging you to put this back on your radar, and wondering whether those changes are in fact coming still that will require this documentation? It's fine to let this sit as a draft, but I did want to make sure we know what's happening here.  :)

Hi @sarah11918 ! I have updated my PR, let's see if this gets merged!


Upgrade prompt_toolkit to 0.52.

@eliangcs This is an upgrade of the prompt toolkit library to 0.52. I'd highly encourage you to test this PR thoroughly before merging.
Changelog for prompt_toolkit (https://github.com/jonathanslenders/python-prompt-toolkit/blob/master/CHANGELOG).
Here are the pending PRs for pgcli and mycli for this exact same change.
pgcli - https://github.com/dbcli/pgcli/pull/362
mycli - https://github.com/dbcli/mycli/pull/159

I had to disable the mouse support since it made it impossible to copy the output.

Don't merge this one. There is a bug in prompt_toolkit where the prompt has extra characters and the enter key seems broken in windows.

Sorry for late reply. Although you have discard the branch, I tested it using behave on Mac and it didn't pass.

Yes there was a bug in prompt_toolkit that was adding a few extra characters in the prompt.


File browser - Download multiple files at once

Hi,
As far as I'm aware, it's not possible to download multiple files at once. It would be nice to be able to multi-select files in the file browser and download a zip file. It's especially useful if you want to download the content of a parquet directory for example, to get the data and meta-data at once.
I'm not a python guy, but if you point me in the right direction in where the code is and the general idea of how you'd do it, I'll try to send a PR with the enhancement.
Thanks,
François

One problem with that is that a Web server does not have much processing power for doing that:
https://issues.cloudera.org/browse/HUE-1112
A better solution would be to have a task server but that's a big new dependency.
Even better might be to run a job under the hood to create this new zip on the grid (that way it will scale) and it won't require any big depdency (just a job template)

@sai-krish FYI, after upload archive, would be a nice one

As Hue uses Knockout.js under the covers, you can run the following code from your browser's console to download all files on the given page:
for(var i = 0; i < viewModel.files().length; i++){
     if(viewModel.files()[i].type === "file"){
          window.open("/filebrowser/download=/" + viewModel.files()[i].path);
     }
}

Similar request from my side, if you are lucky and your files have an index in their pattern use browser "copy as curl"  and then script via bash. Otherwise if your files have very dynamic patterns you need to add another script step to fetch the list of file-names.
If files are too large for stdout then use wget or curl towards are file.
 for i in {0..99}; do curl "https://hue-endpoint.com/filebrowser/download=/your-path/part-0000{$i}"; done >> /tmp/content

Maybe a v1 with Joseph idea, then an Oozie batch job to compress everything into a single zip?

@manuelAldana I tried curl syntax mentioned in your solution, but no matter what option I specify, it either ends up giving me a 302 (tried -L option to resolve it) or ends up dumping javascript code for Hue welcome page. Appreciate if you could give me any pointers as to what i might be doing wrong.
Commands I have tried
curl --cacert <cert_file> "https://hue-endpoint.com/filebrowser/download=//<file_name>" --verbose
gives -> HTTP/1.1 302 FOUND
curl -L --cacert <cert_file> "https://hue-endpoint.com/filebrowser/download=//<file_name>" --verbose
this prints the javascript for Hue home page

I would also like to download folders. My schools research cluster only allows us to use Hue, not SSH.
So thats the only way we can interface with it.

Currently, there is a 'Compress' action that will zip multiple files or folders together, so that they can be downloaded in one file (similarly to Google Drive)

Thanks, I'm having difficulty finding documentation on the "compress" action in hue. Can you point me in the right direction? Thanks

In the filebrowser, if the option enable_extract_uploaded_archive is
enabled (it is by default), you can select a file and from the actions
dropdown, you can select the compress option.
On Tue, Aug 21, 2018 at 6:50 AM Gabriel Fair<EMAIL_ADDRESS>wrote:

Thanks, I'm having difficulty finding documentation on the "compress"
action in hue. Can you point me in the right direction? Thanks
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/cloudera/hue/issues/208#issuecomment-414680941, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AHCQz974C-C7DtGAsVuJwHnYPh6L7b4gks5uTBBCgaJpZM4FZt7k
.

Hi @jdesjean I am using latest Cloudera 5.15.1 and when I try to do compress on a directory this is what I get in my Oozie logs:
>>> Invoking Shell command line now >>

Stdoutput Created temporary output directory: /tmp/tmp.OB54gSmukf
Stdoutput Deleted temporary output directory: /tmp/tmp.OB54gSmukf
Exit code of the Shell command 1
<<< Invocation of Shell command completed <<<

<<< Invocation of Main class completed <<<

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]

Oozie Launcher failed, finishing Hadoop job gracefully

Oozie Launcher, uploading action data to HDFS sequence file: hdfs://multivac-hadoop-master-1:8020/user/mpanahi/oozie-oozi/0000004-180827220407743-oozie-oozi-W/shell-19a1--shell/action-data.seq
Successfully reset security manager from org.apache.oozie.action.hadoop.LauncherSecurityManager@7876d598 to null

Oozie Launcher ends

Hi @jdesjean I am using latest Cloudera 5.15.1 and when I try to do compress on a directory this is what I get in my Oozie logs:
>>> Invoking Shell command line now >>

Stdoutput Created temporary output directory: /tmp/tmp.OB54gSmukf
Stdoutput Deleted temporary output directory: /tmp/tmp.OB54gSmukf
Exit code of the Shell command 1
<<< Invocation of Shell command completed <<<

<<< Invocation of Main class completed <<<

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]

Oozie Launcher failed, finishing Hadoop job gracefully

Oozie Launcher, uploading action data to HDFS sequence file: hdfs://master:8020/user/maziyar/oozie-oozi/0000004-180827220407743-oozie-oozi-W/shell-19a1--shell/action-data.seq
Successfully reset security manager from org.apache.oozie.action.hadoop.LauncherSecurityManager@7876d598 to null

Oozie Launcher ends

Thanks @jdesjean for the respond.
I am actually compressing it in the same directory which is my own directory with full permissions. Also, I copied those logs from the job logs. I will take another look at it, thanks again.

As Hue uses Knockout.js under the covers, you can run the following code from your browser's console to download all files on the given page:
for(var i = 0; i < viewModel.files().length; i++){
     if(viewModel.files()[i].type === "file"){
          window.open("/filebrowser/download=/" + viewModel.files()[i].path);
     }
}

It is also possible to brow through folders and download the contents as you described?


⚠ Bofrost.at has degraded performance

In 11556de, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1800 ms

Resolved: Bofrost.at performance has improved in 5098ddd after 6 minutes.