Workload transfer

Hi, can you explain in summary how would I start a job on one computer and then upload the results from another?
I mean transferring the downloaded dataset to a central GPU unit for filtering, and I would like to upload the final result to your server directly from the GPU unit instead of going back to the original computer.
While for a more generic approach of the 2 steps it will be required for your server to manage the half jobs and perform distribution of workload based on node types, I want to try first to deploy my own droplets for scraping and move inference to my home GPU pc. Later the code can be adapted for mass usage.
So I am looking how to pass a job from one crawlingathome client to another one.

Hi, I'm actually currently working on integrating this into the server update v2.2.0. From that update, there will be 3 types of workers:

Stage 1 - CPU only workers
Stage 2 - GPU only workers
Dual - Workers that do both (currently all are dual)

However, the Stage 1 -> Stage 2 transition is something that needs working on, and may cause this update not to work. The stage one workers somehow need to host the images for the stage 2 workers to use.

So I am looking how to pass a job from one crawlingathome client to another one.

This is actually currently achievable, although rather hacky. I will probably add improved support for this on my to-do list.
Take a look here. You'd want to make a duplicate of this class, but change the __init__ function to the following:
def __init__(self, url, token, shard, start_id, end_id, shard_piece):
    self.s = session() # requests.session instance
    self.url = url
    self.token = token
    self.shard = shard
    self.start_id = start_id  # by default usually class np.int64
    self.end_id = end_id      # ^^^
    self.shard_piece = shard_piece

You can find all these variables using your old client, e.g. client.url will return the url required for this __init___.
You can then interact using the normal functions, acting as the original worker. Please note, however, that workers have a 90 minute timeout. If this is an issue for you, perhaps you could do this so that it run the client.log("Waiting for Stage Two") on all your workers every 30 mins until you run it on your GPU?
Hopefully this should work for the time being. Please let me know if you encounter any issues!

I am doing a variation of this here https://github.com/rvencu/crawlingathome-worker/tree/dev (this is completely untested)
I start from GPU and I want to integrate workers setup from the main script, this way I can control a swarm of them that will start jobs and transfer workload for GPU. I am transferring back the results and they complete your server communication.
I am using a file semaphore mechanism to signal when images were downloaded, and the GPU is polling the workers for completion.
This is for someone that wants to max the use of its GPU during a timeframe and also pay for the workers.
I estimate that 500 euro contribution for workers can take me as far as 5% of the entire project in some 40 days. Of course 40 days of 90% GPU workload at home is in addition to that.
When I am done with that I will use your info to create a version where workers and GPU cooperate but unfortunately for them to communicate we need to disclose their IP addresses. If not, then a centralized hub still needs to be used.

Sounds great! Let me know if you'd like me to help make a server that can create this CPU -> GPU worker communication :)

Yes please.
I envision 3 type of clients (maybe make 3 clients)

scraping worker. This will run scraping part and upload 1 csv + images as a zip file (somewhere). You mark stage completion and some kind of way to serve the zip file on request
GPU worker. this will get a new GPU job from the pending zip files sent by a scraper. download zip and upload final result, mark whole job done
normal client - keep the existing client for individuals wanting to contribute on their own cpu / gpu computer one job at a a time

there are some considerations to make re. renaming the zip files with random names (save them in job properties) so they cannot be altered by malicious users, etc.

Definitely what I envisioned too. I believe some sort of overlay server should be best for this, as I'd like the main server to be kept as simple as possible.
Here's my idea:

CPU only workers create their own client, and begin downloading the files etc.
The GPU worker(s) run an 'overlay server', where they keep the CPU clients alive and manage each job's completion on the GPU.
Once all is done, the job can be marked as done through the GPU worker ("overlay server")

This is partly why I added features that can help with this:
dump_json = cah.dump(client)

Dumps a worker, so that it can be sent as JSON data to the GPU overlay server.

client = cah.load(**dump_json)

Here is where the GPU sever re-loads the client and can now finish the jobs.

The server can also

Sorry - misclick! 😅

What I have against this overlay server is that it should not be GPU bound. We need an independent overlay server. Because people will pop in and out with colab gpus and kaggle gpus at anytime. they are not reliable to keep scrapers alive when they suffer their own sudden deaath

Fair enough. The more that I think of it, it appears the central server can do all that's needed and more compared to overlay servers.
Here's what I'm thinking now:

Clients upload data to a cloud service (there are some pretty good free temporary hosting platforms that definately get the job done)
The client communicates with the main server, submitting its data download URL.
As the central server also runs the timeout feature, the clients can be separated out, and can be listed as the 3 different types you mentioned before.
The GPU client connects, matches and completes the shards - without needing to host any server.

What do you think about this idea? :)

Yes, basically this is it. The challenge would be that we have too many scrapers and too few GPUs, then the payload in the temp sharing site can expire. In such cases GPU node trying to access it should report back to CAH server to unmark the shard for stage 1
a more resilient solution for storage should minimize the potential waste of scrapers

Yeah. As long as it's a URL that can download a ZIP/TAR, It should be fine - so there's room for customisation there.
No promises, but I'll try to attempt to build a system that can do this :)

I was looking for file transfer sites with speed and retention. It is hard to reconcile cost, speed, disk space and reliability. Wetransfer has retired public API. Dropbox has good API and reliability and a plus plan gives 2TB of storage (or about 4000 shards completed after stage 1)
But looking at the scaling aspects there are a lot of potential problems that arise, either with Dropbox or really anyone else in case we want to use 1000s of scrapers at once
A good read here https://developers.dropbox.com/dbx-performance-guide

https://pixeldrain.com/
has API, does not require an account to upload and download, only to delete
10000 zip files in principle (though no account means every worker can upload up to 10000 files)
100 days to keep
not sure about speed capping needs testing

Just a thought, could we set up an SFTP server for the CPU & GPU clients to communicate? And if security isn't needed we could just use FTP.

@rvencu https://tmp.ninja/ seems okay? We might want to consider emailing devs first just to ensure they have the resources ready, but theoretically, any file hosting service should work as long as we have a direct download URL the workers can use to pull the data (even gdrive has this).

@ARKseal It might be better just to integrate it directly into the main server though - also helps with monitoring workers etc.

I agree but beware of the file size. 500MB each shard as result of stage 1

untested yet but very simple, the server just need to store / handle these ids
import requests
import re

class Pixeldrain():
    def __init__(self, api_key=""):
        session = requests.Session()
        self.session = session
        self.key=api_key
        self.url = "https://pixeldrain.com/api/file/"
    def post_file(self, file):
        payload={}
        files=[
        ('file',(file,open(file,'rb'),'application/octet-stream'))
        ]
        headers = {}
        response = requests.request("POST", self.url+'?file', headers=headers, data=payload, files=files)
        info = response.json()
        return info.id
    def get_file(self, id):
        headers = {}
        response = requests.request("GET", self.url+id, headers=headers)
        d = response.headers['content-disposition']
        fname = re.findall("filename=(.+)", d)[0]
        with open(fname, "wb") as file:
            file.write(response.content)

I tried to upload 400MB file and it crashed. I read that requests does not handle HTTP 100 - continue that perhaps happens for such a big file

Did you stream your upload?

did not, of course, so my chances were nil from the start :)

I did stream this time but same error:
from requests_toolbelt.multipart import encoder
import re

class Pixeldrain():
    def __init__(self, api_key=""):
        session = requests.Session()
        self.session = session
        self.key=api_key
        self.url = "https://pixeldrain.com/api/file/"
    def post_file(self, file):
        with open(file,'rb') as f:
            form = encoder.MultipartEncoder({
                "documents": (file, f, "application/octet-stream"),
                "composite": "NONE",
            })
            headers = {"Prefer": "respond-async", "Content-Type": form.content_type}
            response = self.session.post(self.url+'?file', headers=headers, data=form)
            info = response.json()
            return info.id
    def get_file(self, id):
        headers = {}
        response = self.session.get("GET", self.url+id, headers=headers)
        d = response.headers['content-disposition']
        fname = re.findall("filename=(.+)", d)[0]
        with open(fname, "wb") as file:
            file.write("./save/"+response.content)

if __name__ == "__main__":

    pixel = Pixeldrain()

    id = pixel.post_file("crashingset.zip")

    print(id)

    pixel.get_file(id)```

and error is

File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 340, in _send_until_done
return self.connection.send(data)
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1737, in send
self._raise_ssl_error(self._ssl, result)
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1639, in _raise_ssl_error
raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (32, 'EPIPE')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.8/http/client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1301, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1250, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1049, in _send_output
self.send(chunk)
File "/usr/lib/python3.8/http/client.py", line 971, in send
self.sock.sendall(data)
File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 351, in sendall
sent = self._send_until_done(
File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 346, in _send_until_done
raise SocketError(str(e))
OSError: (32, 'EPIPE')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen
retries = retries.increment(
File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 400, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/rvencu/.local/lib/python3.8/site-packages/six.py", line 702, in reraise
raise value.with_traceback(tb)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.8/http/client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1301, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1250, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1049, in _send_output
self.send(chunk)
File "/usr/lib/python3.8/http/client.py", line 971, in send
self.sock.sendall(data)
File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 351, in sendall
sent = self._send_until_done(
File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 346, in _send_until_done
raise SocketError(str(e))
urllib3.exceptions.ProtocolError: ('Connection aborted.', OSError("(32, 'EPIPE')"))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "pixeldrain.py", line 33, in
id = pixel.post_file("crashingset.zip")
File "pixeldrain.py", line 18, in post_file
response = self.session.post(self.url+'?file', headers=headers, data=form)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 581, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', OSError("(32, 'EPIPE')"))

Yeah, I suppose. As long as we can have a direct download URL, it should be fine.

We could theoretically use the server, however, I think it'd be a much better solution to avoid storing it on the server to avoid slowing down the workers.

As of now we have Dropbox as our only possible storage host, and for the plus plan we need to pay $10 usd, not sure if it's cheaper for you. But then again we could use GDrive.

The thing is, the server only has 100GB of storage, and I don't really want to be congesting the server with files if it can't handle the number of requests. I'm going to look into setting up a server built specifically for this crawling@home image hosting, hopefully using a google cloud trial again.

the protocol should first attempt peer connectivity between scraper and gpu worker. in case it fails then save the files at the transfer server...

A great idea. I wonder how we could pull this off without having to forward ports?

look here https://github.com/magic-wormhole/magic-wormhole

I tested this out:
We just need the inference worker's ip.
Reciever Script
`import socket
import tqdm
import os
SERVER_HOST = "<IP_ADDRESS>"
SERVER_PORT = 5001
BUFFER_SIZE = 4096
SEPARATOR = ""
s = socket.socket()
s.bind((SERVER_HOST, SERVER_PORT))
s.listen(5)
print(f'[*] Listening as {SERVER_HOST}:{SERVER_PORT}')
client_socket, address = s.accept()
print(f'[+] {address} is connected')
received = client_socket.recv(BUFFER_SIZE).decode()
filename, filesize = received.split(SEPARATOR)
filename = os.path.basename(filename)
filesize = int(filesize)
progress = tqdm.tqdm(range(filesize), f'Recieving {filename}', unit='B', unit_scale=True, unit_divisor=1024)
with open(filename, 'wb') as f:
while True:
bytes_read = client_socket.recv(BUFFER_SIZE)
if not bytes_read:
break
f.write(bytes_read)
progress.update(len(bytes_read))
client_socket.close()
s.close()`
Sender Script:
`import socket
import tqdm
import os
SEPARATOR = ""
BUFFER_SIZE = 4096
receiver ip
host = ""
port = 5001
filename = "test.zip"
filesize = os.path.getsize(filename)
s = socket.socket()
print(f'[+] Connecting to {host}:{port}')
s.connect((host, port))
print('[+] Connected')
s.send(f'{filename}{SEPARATOR}{filesize}'.encode())
progress = tqdm.tqdm(range(filesize), f'Sending {filename}', unit='B', unit_scale=True, unit_divisor=1024)
with open(filename, 'rb') as f:
while True:
bytes_read = f.read(BUFFER_SIZE)
if not bytes_read: break
s.sendall(bytes_read)
progress.update(len(bytes_read))
s.close()`

That looks great! Just a couple questions:

What happens if a CPU worker was to go offline?
How could we query a certain shard's file / by a certain filename?

I do not want to expose my GPU box to the internet.

@rvencu I believe that magic-wormhole also stores your IP, allowing for the transfer. Maybe we can use Tor to provide security?

Seems that it supports Tor. Or not?
https://magic-wormhole.readthedocs.io/en/latest/tor.html#onion-servers


chore: fix keep-sorted linting issue

Description
Sort the services array correct again.
This will prevent the following issue with lint:
nr lint

> @ lint /Users/ab/dev/forks/vscode-file-nesting-config
> eslint .

(node:98618) [DEP0180] DeprecationWarning: fs.Stats constructor is deprecated.
(Use `node --trace-deprecation ...` to show where the warning was created)

/Users/ab/dev/forks/vscode-file-nesting-config/update.mjs
  45:18  error  [keep-sorted] fix: Keep sorted  command/command

✖ 1 problem (1 error, 0 warnings)
  1 error and 0 warnings potentially fixable with the `--fix` option.

 ELIFECYCLE  Command failed with exit code 1.

Linked Issues
no issue to link
Additional context
no additional context to mention

Thank you, it's included in #245 tho.


misc: make copyUntypedFiles work for watch mode

Pre-flight checklist

[x] I have read the Contributing Guidelines on pull requests.
[ ] If this is a code change: I have written unit tests and/or added dogfooding pages to fully verify the new behavior.
[ ] If this is a new API or substantial change: the PR has an accompanying issue (closes #0000) and the maintainers have approved on my working plan.

Motivation
The last missing bit of DX. copyUntypedFiles.mjs now work for watch mode as well, with a simple chokidar watcher.
Test Plan
We can now hot-reload CSS as well! For example, here let's pretend we are editing Details/styles.module.css for theme-common (which used to not work):

Test links

Deploy preview: https://deploy-preview-_____--docusaurus-2.netlify.app/
Related issues/PRs

great 👍 thanks


[6.0][Tests] NFC: Gate preconcurrency conformances runtime test on presenc…

…e of concurrency runtime
Resolves: rdar://123810657
(cherry picked from commit 34f96242fed5003650a375121e8eff0b2c677ab4)

@swift-ci please test


Support GVRP on selected Networks and NICs

Fixes #8318
Signed-off-by: Tim Rots<EMAIL_ADDRESS>

This pull request didn't trigger Jenkins as its author isn't in the whitelist.
An organization member must perform one of the following:

To have this branch tested by Jenkins, use the "ok to test" command.
To have a one time test done, use the "test this please" command.

Those commands are simple Github comments of the format: "jenkins: COMMAND"

This pull request didn't trigger Jenkins as its author isn't in the whitelist.
An organization member must perform one of the following:

To have this branch tested by Jenkins, use the "ok to test" command.
To have a one time test done, use the "test this please" command.

Those commands are simple Github comments of the format: "jenkins: COMMAND"

Hi @tomponline , As you assigned the above referenced issue. Would you like to help with some reviews to get this feature right?

Hi @tomponline , As you assigned the above referenced issue. Would you like to help with some reviews to get this feature right?

@TimRots great thanks, did the feature work as original poster intended? (I've got an old switch at home that supports GARP/GRVP apparently, but not had change to set it up yet).

@TimRots great thanks, did the feature work as original poster intended? (I've got an old switch at home that supports GARP/GRVP apparently, but not had change to set it up yet).

Thank you for the review @tomponline, will update the PR later today with your suggestions.

Thank you for the review @tomponline, will update the PR later today with your suggestions.

So the plumbing of this feature looks good now (with a couple of small code style nits).

Just committed the requested changes from your last review. Looks much better :)

However we also need to allow the gvrp config property to be set in each of the affected NICs and Network drivers.

Thanks for the clear explanation. Working on it.

So the plumbing of this feature looks good now (with a couple of small code style nits).

Just committed the requested changes from your last review. Looks much better :)

However we also need to allow the gvrp config property to be set in each of the affected NICs and Network drivers.

Thanks for the clear explanation. Working on it.

Thank you for another round of review.

Is it worth also adding the new field to the validation rules for routed, physical and ipvlan NICs too (like you did for macvlan nic) so they can all benefit (and update associated docs). And update the API extension to indicate which NICs the new field is available for.

That definitely makes sense to do. Will update the PR and ask for your review again.

Do you have a GVRP compatible switch to test this?

(Unfortunately) I don't have one available for testing. Would it be possible for you to help out to test the feature?

Thank you for another round of review.

Is it worth also adding the new field to the validation rules for routed, physical and ipvlan NICs too (like you did for macvlan nic) so they can all benefit (and update associated docs). And update the API extension to indicate which NICs the new field is available for.

That definitely makes sense to do. Will update the PR and ask for your review again.

Do you have a GVRP compatible switch to test this?

(Unfortunately) I don't have one available for testing. Would it be possible for you to help out to test the feature?

Looking good at first sight. Would you mind splitting things in 3 commits?

api: network_gvrp extension

Include the shared/api and doc/api-extensions.md bits in there

doc: Support gvrp on macvlan

The doc/instance.md and doc/networks.md bits

lxd: Support gvrp on macvlan'

All the lxd/* bits (should be everything else)

That would match our usual structure and helps with handling of the stable branches.
Thanks!

Looking good at first sight. Would you mind splitting things in 3 commits?

api: network_gvrp extension

Include the shared/api and doc/api-extensions.md bits in there

doc: Support gvrp on macvlan

The doc/instance.md and doc/networks.md bits

lxd: Support gvrp on macvlan'

All the lxd/* bits (should be everything else)

That would match our usual structure and helps with handling of the stable branches.
Thanks!

@stgraber will split it up in 3 separate pr's as requested. Could you however run Jenkins once on this pr? Thanks in advance.

@stgraber will split it up in 3 separate pr's as requested. Could you however run Jenkins once on this pr? Thanks in advance.

Oh, I meant commits in the same PR :)

Oh, I meant commits in the same PR :)

We still want to merge it in one shot, so one PR but 3 commits in the PR.

We still want to merge it in one shot, so one PR but 3 commits in the PR.

@stgraber I'd like to test this on a GVRP compatible switch too before merging as thus far its not been tested working yet (beyond compiling). I believe my Netgear GS108T supports this, but its downstairs so need to test this on my laptop.

@stgraber I'd like to test this on a GVRP compatible switch too before merging as thus far its not been tested working yet (beyond compiling). I believe my Netgear GS108T supports this, but its downstairs so need to test this on my laptop.

Thanks. I think we are nearly there with this. One tweak to the api-extensions.md file and splitting the commits as @stgraber suggested. Then I just need to check the gvrp argument to ip link actually works as expected and we're done.

Thanks @tomponline and @stgraber for the help. I will split the PR into 3 commits as requested later today.

Thanks. I think we are nearly there with this. One tweak to the api-extensions.md file and splitting the commits as @stgraber suggested. Then I just need to check the gvrp argument to ip link actually works as expected and we're done.

Thanks @tomponline and @stgraber for the help. I will split the PR into 3 commits as requested later today.

@TimRots looks like you've got a conflict with master.

@TimRots looks like you've got a conflict with master.

I've resolved the merge conflict.
However I cannot test this as non of my switches support GARP/GVRP, the GS108Tv3 has it, but I have the GS108Tv2.
I've asked the OP if they can help test it https://discuss.linuxcontainers.org/t/placing-containers-in-vlans-with-gvrp/9891/7
@TimRots are you able to split this into 3 commits or are you happy for me to do it?

I've resolved the merge conflict.
However I cannot test this as non of my switches support GARP/GVRP, the GS108Tv3 has it, but I have the GS108Tv2.
I've asked the OP if they can help test it https://discuss.linuxcontainers.org/t/placing-containers-in-vlans-with-gvrp/9891/7
@TimRots are you able to split this into 3 commits or are you happy for me to do it?

jenkins: test this please

jenkins: test this please

Testsuite passed

Testsuite passed

I've resolved the merge conflict.
However I cannot test this as non of my switches support GARP/GVRP, the GS108Tv3 has it, but I have the GS108Tv2.
I've asked the OP if they can help test it https://discuss.linuxcontainers.org/t/placing-containers-in-vlans-with-gvrp/9891/7
@TimRots are you able to split this into 3 commits or are you happy for me to do it?

@tomponline thank you for stepping in and sorry for my late reply. I did not have time yet this week to untangle this, due to time/work.
Would you be able to split/test the PR as needed, I will only be able to do this in the weekend or thereafter.

I've resolved the merge conflict.
However I cannot test this as non of my switches support GARP/GVRP, the GS108Tv3 has it, but I have the GS108Tv2.
I've asked the OP if they can help test it https://discuss.linuxcontainers.org/t/placing-containers-in-vlans-with-gvrp/9891/7
@TimRots are you able to split this into 3 commits or are you happy for me to do it?

@tomponline thank you for stepping in and sorry for my late reply. I did not have time yet this week to untangle this, due to time/work.
Would you be able to split/test the PR as needed, I will only be able to do this in the weekend or thereafter.

@stgraber I've rebased and splits the commits, but kept @TimRots attribution in the commits.

@stgraber I've rebased and splits the commits, but kept @TimRots attribution in the commits.

jenkins: test this please

jenkins: test this please

jenkins: test this please

jenkins: test this please

Testsuite passed

Testsuite passed


Adding DFD file (Issue #14)

Added DFD file and a folder of "documents" that would hold all the relevant documents for our project.

Issue link:
https://github.com/redhat-beyond/MeetBalls/issues/14

I took a look, and it looks great and detailed.
Good job, well done.

It looks great for me.
Good job!

@ZvikaNaorCohen the issue would have been closed automatically if you mentioned it in the first comment (the PR description) along with one of the keywords mentioned here: https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword


feat: Virtualized docs list, auto scroll to active item on refresh

Three main things being added here

virtualizing the list of items in the docs list using tanstack react virtualizer
when user refreshes the page, if the active item is not visible in the docs list, list will be scrolled until the active item is visible
added a button so if you scroll away from the currently active item and want to return to it, it will be easy to do so

==
Video showing page refresh results in active item being visible, and the "scroll to" button when it is scrolled out of frame:
https://github.com/TobikoData/sqlmesh/assets/81378/469b7318-dd77-4d34-9865-9d97a26b8488

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Use 21.0w for SPARK verification in automated tests

21.0w should be additionally used.

This version is of no relevance anymore.


Add support for deleted files

Right now the comment that is generated by this action splits the files into 3 different sections

files that got bigger
files that got smaller
files that stayed the same size

This misses the last possible classification files that were deleted and you can see that in the code we have a TODO to cover this case: https://github.com/mainmatter/ember-asset-size-action/blob/master/lib/helpers.mjs#L46
Deleted files should be tracked in a new distinct section so that they are explicitly deleted and not just that they got smaller by 100% 👍

This is fixed since https://github.com/mainmatter/ember-asset-size-action/pull/41 was merged


Hide search when syncing is in process

Implemented at https://github.com/empratyush/Apps/commit/37d885c2c4d46cf07e2d205c54db2e9108c7e306


Playground error when "Generate key and publish introduction"

After logging in with ORCID, the playground is throwing the following error
Error publishing Nanopub Introduction: Error sendind the HTTP request: error sending request
after clicking on the Generate key and publish introduction button.
Chrome console shows a CORS error:

This was repo'd with with two different users/ORCID's.

Here's a video showing the whole Network tab
https://www.loom.com/share/3ec3e0f138824da8b412d33ef8263a2c

OK, thanks. I could reproduce this.
It seems to require a Access-Control-Allow-Headers header. I am not sure why this is now necessary. It used to work, so I think it wasn't needed previously.
Anyway, I added this header line to the relevant servers, and I think it should work now. Could you try again?

Fixed, thanks!


Refactor FeatFinderCLI and add unittests

FeatFinderCLI is one of the scripts included in RDkit. This PR refactors the script by switching from getopt to argparse and separating the parser and execution of the code. This allows writing a Unittest for the script.

Thanks Peter.


Push a new release to PyPI

Would it be possible to create a new release and push it up to PyPI? Thanks!

Sorry for the late reply. I should have time to push a new release this week.

Pushed release for 0.1.1 to PyPI


Indirect Call labeling

Hi,
I am wondering why the GHIDRA decompiler is naming globally defined variables as pcRam0000000000000000 as opposed to say DAT_00011cc8 when doing an indirect call, after running the default set of analyzers.
Here is an example of my indirect call using a pointer located at 0x11cc8 being renamed to *pcRam000000000000000 as opposed to (**DAT_00011cc8)( DAT_00011cc8,local_res8). In most cases, GHIDRA seems to display the latter, but id say in ~10% of binaries I look at, I get the former, and i'm not sure why...
Is one of the default analyzers doing this? Is this expected behavior?

Try change permissions (to rx or r) for the memory block, which contains the variable or locally for global variable (Mutability setting to constant). That helped?

Giving the memory block write permissions seems to fix the problem. Thanks!
Do you know why this is happening? In one binary, I have several globally defined pointers  in the .text section being used for indirect calls, but only some of them get decompiled as *pcRam0000000000.....
Anyhow, giving the .text write permissions seems to fix my problem, so at least I have a workaround! Thanks again.


Angular 14 doesn't build library with multiple entry points

Which @angular/* package(s) are the source of the bug?
compiler-cli, compiler, upgrade
Is this a regression?
Yes
Description
Hi everyone,
I am about to migrate from v13 to v14, but I faced with the following issue.
When I try to build library project it fails on the first entry point which has reference to another entry with the error:
------------------------------------------------------------------------------
Building entry point 'xxx'
------------------------------------------------------------------------------
✖ Compiling with Angular sources in Ivy partial compilation mode.
PATH_TO_TS - error TS2305: Module '"yyy"' has no exported member 'xyz'.

But the thing is that everything is in place and it works fine with ng v13. Moreover ng serve is working fine, but not the build.
I created a repo for this issue, main branch contains working version (ng v13) and ng-14 branch has the same code after ng update @angular/core@14 @angular/cli@14 with the issue.
To reproduce the issue.

Clone main branch
Make sure that it builds the lib by running npm run build-components (ng build components)
Update angular to v14 by  running ng update @angular/core@14 @angular/cli@14
Build component library npm run build-components

Result: failed to build
Any thoughts?
Please provide a link to a minimal reproduction of the bug
https://github.com/mgameover/app-n-lib/tree/ng-14
Please provide the exception or error you saw
> ng build components

------------------------------------------------------------------------------
Building entry point '@my/components'
------------------------------------------------------------------------------
✔ Compiling with Angular sources in Ivy partial compilation mode.
✔ Writing FESM bundles
✔ Copying assets
✔ Writing package manifest
✔ Built @my/components

------------------------------------------------------------------------------
Building entry point '@my/components/shared'
------------------------------------------------------------------------------
✔ Compiling with Angular sources in Ivy partial compilation mode.
✔ Writing FESM bundles
✔ Built @my/components/shared

------------------------------------------------------------------------------
Building entry point '@my/components/component'
------------------------------------------------------------------------------
✖ Compiling with Angular sources in Ivy partial compilation mode.
projects/components/component/components.component.ts:3:10 - error TS2305: Module '"@my/components/shared"' has no exported member 'isDefined'.

3 import { isDefined } from '@my/components/shared';

Please provide the environment you discovered this bug in (run ng version)
Angular CLI: 14.0.0
Node: 16.14.2
Package Manager: npm 8.5.5
OS: win32 x64

Angular: 14.0.0
... animations, cli, common, compiler, compiler-cli, core, forms
... platform-browser, platform-browser-dynamic, router

Package                         Version
---------------------------------------------------------
@angular-devkit/architect       0.1400.0
@angular-devkit/build-angular   14.0.0
@angular-devkit/core            14.0.0
@angular-devkit/schematics      14.0.0
@schematics/angular             14.0.0
ng-packagr                      14.0.0
rxjs                            7.5.5
typescript                      4.6.4

Anything else?
No response

The cause of the problem here is that there is an index file (https://github.com/mgameover/app-n-lib/blob/ng-14/projects/components/shared/index.ts ) which exports all the symbols but it is not mark as the entry-point.
This causes a recursive exports of itself, as now in version 14 when the entry-point is something other than an index.ts an index file is generated.
We should handle this case. As a workaround for the time being, you can move the contents of the index in the public-api file.

It looks like the path mapping is pointing to the source of the library and not the built version in the dist.

yes, path mapping points to source and that's done intentionally - changes in the lib are picked up automatically so you don't need to rebuild lib everyone.

Hi @alan-agius4,
As I can see from the changelog, the fix has been released in v14.0.1.
However I don't see any changes. I tried to update the same repo, I can see that the version is changed to v14.0.1 but it doesn't solve the issue.
So my question is  - what is a proper way to handle this?

Remove index.ts
or rename public-api.ts to index.ts and update ng-package.json
or there will be another fix

In your particular case the best action would be to rename public-api.ts to index.ts and update ng-package.json

It actually looks like 14.0.1 didn't contain the change mentioned in the
ng-packagr. I will cut another ng-packagr release later this week.
On Fri, 10 Jun 2022 at 16:05, Alex @.***> wrote:

Hi @alan-agius4 https://github.com/alan-agius4,
As I can see from the changelog
https://github.com/ng-packagr/ng-packagr/blob/main/CHANGELOG.md, the
fix has been released in v14.0.1.
However I don't see any changes. I tried to update the same repo, I can
see that the version is changed to v14.0.1 but it doesn't solve the issue.
So my question is - what is a proper way to handle this?

Remove index.ts
or rename public-api.ts to index.ts and update ng-package.json
or there will be another fix

—
Reply to this email directly, view it on GitHub
https://github.com/angular/angular-cli/issues/23293#issuecomment-1152399068,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEF74WTJIXHW2IFUGP3GI3LVONDTDANCNFSM5X34YVVA
.
You are receiving this because you were mentioned.Message ID:
@.***>

Ng-packagr version 14.0.2 has been released.

For anyone coming across this with a slightly different error
Entry point 'my-lib' has an 'index.ts' parallel to the 'entryFilePath'. The 'entryFilePath' should be updated to point to the 'index.ts' file.

Make sure you don't have internal imports in the library, that point to the main index.ts file of the lib.


Consider droping PHP 5 and 7.0.

PHP Roadmap.

http://php.net/supported-versions.php

Thanks!
We don’t just drop support for PHP versions as soon as we can. We only do so if some version holds us back or is really not used anymore in practice.
So are there any specific reasons for this request?

I just strongly recommend it considering why support PHP 5 syntax when it should not be used. Plus if you support PHP 5 you can not use any of PHP >= 7.1 features. It just seems to be a time waster to support PHP 5 and 7.0. What i suggest is the next major version just drop PHP 5 and PHP 7.0.

Well, yes, but the whole library is already compliant with the old syntax today, and PHP 7.0 (and even PHP 5) are definitely still being used outside there. For example, they come pre-packaged with several Linux distributions.
But we’ll drop them as soon as it’s the right time for that, which should be soon.
Thanks again!


ACME provider is agressively retrying in 2.0

Do you want to request a feature or report a bug?
Bug
Did you try using a 1.7.x configuration for the version 2.0?

[ ] Yes
[X] No

What did you do?
Setup a TLS ingress on Traefik v2.0, for some reasons this domain is rate limited by let's encrypt, but Traefik is retrying a lot to get this certificates.
What did you expect to see?
Traefik delaying retries if let's encrypt failed.
What did you see instead?
Traefik agressively retrying.
Output of traefik version: (What version of Traefik are you using?)

Name:         blog-ingress
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  traefik.containo.us/v1alpha1
Kind:         IngressRoute
Metadata:
  Creation Timestamp:  2019-09-17T17:11:03Z
  Generation:          3
  Resource Version:    2039109
  Self Link:           /apis/traefik.containo.us/v1alpha1/namespaces/default/ingressroutes/blog-ingress
  UID:                 412eb69b-cb50-47b8-8dcb-4835793936a7
Spec:
  Entry Points:
    websecure
  Routes:
    Kind:   Rule
    Match:  Host(`blogadmin.xxxxx.com`)
    Middlewares:
      Name:  https-only
    Services:
      Name:  caddy-git
      Port:  80
  Tls:
    Cert Resolver:  default

Logs
 time="2019-09-18T13:07:42Z" level=error msg="Unable to obtain ACME certificate for domains \"blogadmin.xxxx.com\": unable to generate a certificate for the domains [blogadmin.xxxx.com]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many certificates already issued for exact set of domains: blogadmin.xxxx.com: see https://letsencrypt.org/docs/rate-limits/, url: " routerName=default-blog-ingress-eec3963c95099db09e38 rule="Host(`blogadmin.xxxxx.com`)" providerName=default.acme
time="2019-09-18T13:07:53Z" level=error msg="Unable to obtain ACME certificate for domains \"blogadmin.xxxxx.com\": unable to generate a certificate for the domains [blogadmin.xxxx.com]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many certificates already issued for exact set of domains: blogadmin.xxxx.com: see https://letsencrypt.org/docs/rate-limits/, url: " providerName=default.acme routerName=default-blog-ingress-eec3963c95099db09e38 rule="Host(`blogadmin.xxxx.com`)"
time="2019-09-18T13:07:55Z" level=error msg="Unable to obtain ACME certificate for domains \"blogadmin.xxxxx.com\": unable to generate a certificate for the domains [blogadmin.xxxxx.com]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many certificates already issued for exact set of domains: blogadmin.xxxxx.com: see https://letsencrypt.org/docs/rate-limits/, url: " providerName=default.acme routerName=default-blog-ingress-eec3963c95099db09e38 rule="Host(`blogadmin.xxxxx.com`)"
time="2019-09-18T13:07:56Z" level=error msg="Unable to obtain ACME certificate for domains \"blogadmin.xxxxx.com\": unable to generate a certificate for the domains [blogadmin.xxxxx.com]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many certificates already issued for exact set of domains: blogadmin.xxxxx.com: see https://letsencrypt.org/docs/rate-limits/, url: " providerName=default.acme routerName=default-blog-ingress-eec3963c95099db09e38 rule="Host(`blogadmin.xxxxx.com`)"
time="2019-09-18T13:07:56Z" level=error msg="Unable to obtain ACME certificate for domains \"blogadmin.xxxxx.com\": unable to generate a certificate for the domains [blogadmin.xxxxx.com]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many certificates already issued for exact set of domains: blogadmin.xxxxx.com: see https://letsencrypt.org/docs/rate-limits/, url: " routerName=default-blog-ingress-eec3963c95099db09e38 rule="Host(`blogadmin.xxxxx.com`)" providerName=default.acme
time="2019-09-18T13:07:59Z" level=error msg="Unable to obtain ACME certificate for domains \"blogadmin.xxxxx.com\": unable to generate a certificate for the domains [blogadmin.xxxxx.com]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many certificates already issued for exact set of domains: blogadmin.xxxxx.com: see https://letsencrypt.org/docs/rate-limits/, url: " routerName=default-blog-ingress-eec3963c95099db09e38 rule="Host(`blogadmin.xxxxx.com`)" providerName=default.acme
time="2019-09-18T13:07:59Z" level=error msg="Unable to obtain ACME certificate for domains \"blogadmin.xxxxx.com\": unable to generate a certificate for the domains [blogadmin.xxxxx.com]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many certificates already issued for exact set of domains: blogadmin.xxxxx.com: see https://letsencrypt.org/docs/rate-limits/, url: " providerName=default.acme routerName=default-blog-ingress-eec3963c95099db09e38 rule="Host(`blogadmin.xxxxx.com`)"
time="2019-09-18T13:08:03Z" level=error msg="Unable to obtain ACME certificate for domains \"blogadmin.xxxxx.com\": unable to generate a certificate for the domains [blogadmin.xxxxx.com]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many certificates already issued for exact set of domains: blogadmin.xxxxx.com: see https://letsencrypt.org/docs/rate-limits/, url: " providerName=default.acme routerName=default-blog-ingress-eec3963c95099db09e38 rule="Host(`blogadmin.xxxxx.com`)"

You can tell by the timestamps it's probably too often, leading to maybe more rate limiting.
If applicable, please paste the log output in DEBUG level (--log.level=DEBUG switch)

Hi! I'm Træfiker :robot: the bot in charge of communication regulation.
Thanks for your interest in Traefik!
Issue templates help us help you by providing all necessary information.
Please edit your issue and use the available templates:

Feature request
Bug report

And remember: each time someone ignores the template, a cute little bunny dies.

#SaveTheCuteBunny :heart: :rabbit: :heart:

template is respected ?

Hello @akhenakh! Looks like I can try to fix this issue. Could you please add your TLS config to the issue?

Here is one I know it failed with but it happens anytime the letsencrypt api is rate limiting
            - --api.insecure
            - --accesslog
            - --entrypoints.web.address=:8000
            - --entrypoints.websecure.address=:4443
            - --entrypoints.metrics.address=:8888
            - --api.dashboard=true
            - --providers.kubernetescrd
            -<EMAIL_ADDRESS>            - --certificatesresolvers.default.acme.storage=/data/acme.json
            - --certificatesResolvers.default.acme.dnsChallenge.provider=gandiv5
            - --metrics.prometheus=true
            - --metrics.prometheus.entryPoint=metrics
            - --tracing=false
            - --tracing.jaeger=false
            - --tracing.jaeger.localAgentHostPort=jaeger-agent:6831
            - --tracing.jaeger.samplingParam=1
            - --tracing.jaeger.samplingType=const
            - --tracing.jaeger.propagation=b3

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-ingress
  namespace: default
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`traefik.dev.xxxx`)
      middlewares:
        - name: https-compress-admin-auth
      kind: Rule
      services:
        - name: traefik-admin
          port: 8080
  tls:
    certResolver: default

Hello there,
I'm facing this issue, too. It seems that Traefik v2.0 is trying to create a certificate whenever it detects a new HTTP connection to a site without a certificate.
I was able to get SSL certificates by enabling firewall to stop all communications to each site until Traefik generates the new SSL certificate. For example, my sites are monitored with uptimerobot and each healthcheck forced a new petition to LetsEncrypt API asking for the SSL certificate.
Maybe it would be enogh to use a flag or something to check that only one request is in progress per domain name at the same time.
Thanks!

We're also facing this bug unfortunately.
I tried to setup a new router to a client service and tried to use a Host rule with a typo in one of the dns names.
Result: My host is now rate limited.
Is there any way to stop the process which obtains new certificates if there was an error at first try (or even second or third try)?
It seems the current source code does not handle this situation too well.

At least in v2.1, the bug seems still there.
Will soon check again after upgrading to v2.4.

First evidence shows that this might be fixed in v2.4 :-)
I tried starting Traefik with empty acme.json, and then hitting F5 like hell in multiple browser tabs.
I will double-check again, and test the new Traefik version also on CentOS.
Also note that with v2.1, we still used Docker-label-based routing.
With v2.4, we're using a hardcoded routing table instead.
So the better behavior might be also related to that change.
Traefik v2.4 (Ubuntu 20.04):
$ sudo docker logs faro-reverse-proxy | grep -iE  "need ACME|obtained"
time="2021-02-08T13:39:07Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." providerName=default.acme routerName=faro-config-ui-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/setup`)"
time="2021-02-08T13:39:30Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" routerName=faro-config-ui-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/setup`)" providerName=default.acme

Traefik v2.1 (CentOS):
$ sudo docker logs faro-reverse-proxy | grep -iE  "need ACME|obtained"
time="2021-02-08T13:35:01Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." providerName=default.acme routerName=px-server-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/pointexport/`)"
time="2021-02-08T13:35:02Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." providerName=default.acme routerName=server-health-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/server-health`)"
time="2021-02-08T13:35:02Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." providerName=default.acme routerName=dex-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/dex/`)"
time="2021-02-08T13:35:02Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." rule="Host(`ourdomain.example.com`) && PathPrefix(`/setup`)" providerName=default.acme routerName=config-ui-https@file
time="2021-02-08T13:35:03Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." providerName=default.acme routerName=qspage-frontend-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/queue-ui/`)"
time="2021-02-08T13:35:03Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." rule="Host(`ourdomain.example.com`) && PathPrefix(`/queue/`)" routerName=qs-backend-https@file providerName=default.acme
time="2021-02-08T13:35:03Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." providerName=default.acme routerName=ws-backend-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/`)"
time="2021-02-08T13:35:03Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." providerName=default.acme rule="Host(`ourdomain.example.com`) && PathPrefix(`/manual`)" routerName=ourapp-manual-enterprise-https@file
time="2021-02-08T13:35:05Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." providerName=default.acme routerName=ws-backend-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/`)"
time="2021-02-08T13:35:05Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." rule="Host(`ourdomain.example.com`) && PathPrefix(`/dex/`)" routerName=dex-https@file providerName=default.acme
time="2021-02-08T13:35:06Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." rule="Host(`ourdomain.example.com`) && PathPrefix(`/server-health`)" routerName=server-health-https@file providerName=default.acme
time="2021-02-08T13:35:06Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." providerName=default.acme routerName=ourapp-manual-enterprise-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/manual`)"
time="2021-02-08T13:35:06Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." rule="Host(`ourdomain.example.com`) && PathPrefix(`/queue/`)" providerName=default.acme routerName=qs-backend-https@file
time="2021-02-08T13:35:06Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." routerName=config-ui-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/setup`)" providerName=default.acme
time="2021-02-08T13:35:06Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." rule="Host(`ourdomain.example.com`) && PathPrefix(`/pointexport/`)" providerName=default.acme routerName=px-server-https@file
time="2021-02-08T13:35:06Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." providerName=default.acme routerName=qspage-frontend-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/queue-ui/`)"
time="2021-02-08T13:35:17Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" providerName=default.acme routerName=px-server-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/pointexport/`)"
time="2021-02-08T13:35:20Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" providerName=default.acme routerName=qspage-frontend-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/queue-ui/`)"
time="2021-02-08T13:35:26Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" providerName=default.acme routerName=ourapp-manual-enterprise-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/manual`)"
time="2021-02-08T13:35:31Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" providerName=default.acme routerName=ws-backend-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/`)"
time="2021-02-08T13:35:33Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" routerName=dex-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/dex/`)" providerName=default.acme
time="2021-02-08T13:35:33Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" rule="Host(`ourdomain.example.com`) && PathPrefix(`/queue/`)" routerName=qs-backend-https@file providerName=default.acme
time="2021-02-08T13:35:35Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" rule="Host(`ourdomain.example.com`) && PathPrefix(`/pointexport/`)" providerName=default.acme routerName=px-server-https@file
time="2021-02-08T13:35:35Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" routerName=qs-backend-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/queue/`)" providerName=default.acme
time="2021-02-08T13:35:36Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" providerName=default.acme rule="Host(`ourdomain.example.com`) && PathPrefix(`/server-health`)" routerName=server-health-https@file
time="2021-02-08T13:35:39Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" routerName=config-ui-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/setup`)" providerName=default.acme
time="2021-02-08T13:35:39Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" providerName=default.acme routerName=config-ui-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/setup`)"
time="2021-02-08T13:35:39Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" providerName=default.acme rule="Host(`ourdomain.example.com`) && PathPrefix(`/manual`)" routerName=ourapp-manual-enterprise-https@file
time="2021-02-08T13:35:41Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" providerName=default.acme routerName=ws-backend-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/`)"
time="2021-02-08T13:35:42Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" providerName=default.acme routerName=qspage-frontend-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/queue-ui/`)"
time="2021-02-08T13:35:42Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" providerName=default.acme routerName=server-health-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/server-health`)"
time="2021-02-08T13:35:43Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" providerName=default.acme rule="Host(`ourdomain.example.com`) && PathPrefix(`/dex/`)" routerName=dex-https@file
[centos@ip-172-31-12-129 ~]$ sudo docker logs faro-reverse-proxy | grep -iE  "need ACME|obtained"
time="2021-02-08T13:35:01Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." providerName=default.acme routerName=px-server-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/pointexport/`)"
time="2021-02-08T13:35:02Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." providerName=default.acme routerName=server-health-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/server-health`)"
time="2021-02-08T13:35:02Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." providerName=default.acme routerName=dex-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/dex/`)"
time="2021-02-08T13:35:02Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." rule="Host(`ourdomain.example.com`) && PathPrefix(`/setup`)" providerName=default.acme routerName=config-ui-https@file
time="2021-02-08T13:35:03Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." providerName=default.acme routerName=qspage-frontend-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/queue-ui/`)"
time="2021-02-08T13:35:03Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." rule="Host(`ourdomain.example.com`) && PathPrefix(`/queue/`)" routerName=qs-backend-https@file providerName=default.acme
time="2021-02-08T13:35:03Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." providerName=default.acme routerName=ws-backend-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/`)"
time="2021-02-08T13:35:03Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." providerName=default.acme rule="Host(`ourdomain.example.com`) && PathPrefix(`/manual`)" routerName=ourapp-manual-enterprise-https@file
time="2021-02-08T13:35:05Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." providerName=default.acme routerName=ws-backend-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/`)"
time="2021-02-08T13:35:05Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." rule="Host(`ourdomain.example.com`) && PathPrefix(`/dex/`)" routerName=dex-https@file providerName=default.acme
time="2021-02-08T13:35:06Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." rule="Host(`ourdomain.example.com`) && PathPrefix(`/server-health`)" routerName=server-health-https@file providerName=default.acme
time="2021-02-08T13:35:06Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." providerName=default.acme routerName=ourapp-manual-enterprise-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/manual`)"
time="2021-02-08T13:35:06Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." rule="Host(`ourdomain.example.com`) && PathPrefix(`/queue/`)" providerName=default.acme routerName=qs-backend-https@file
time="2021-02-08T13:35:06Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." routerName=config-ui-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/setup`)" providerName=default.acme
time="2021-02-08T13:35:06Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." rule="Host(`ourdomain.example.com`) && PathPrefix(`/pointexport/`)" providerName=default.acme routerName=px-server-https@file
time="2021-02-08T13:35:06Z" level=debug msg="Domains [\"ourdomain.example.com\"] need ACME certificates generation for domains \"ourdomain.example.com\"." providerName=default.acme routerName=qspage-frontend-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/queue-ui/`)"
time="2021-02-08T13:35:17Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" providerName=default.acme routerName=px-server-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/pointexport/`)"
time="2021-02-08T13:35:20Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" providerName=default.acme routerName=qspage-frontend-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/queue-ui/`)"
time="2021-02-08T13:35:26Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" providerName=default.acme routerName=ourapp-manual-enterprise-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/manual`)"
time="2021-02-08T13:35:31Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" providerName=default.acme routerName=ws-backend-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/`)"
time="2021-02-08T13:35:33Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" routerName=dex-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/dex/`)" providerName=default.acme
time="2021-02-08T13:35:33Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" rule="Host(`ourdomain.example.com`) && PathPrefix(`/queue/`)" routerName=qs-backend-https@file providerName=default.acme
time="2021-02-08T13:35:35Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" rule="Host(`ourdomain.example.com`) && PathPrefix(`/pointexport/`)" providerName=default.acme routerName=px-server-https@file
time="2021-02-08T13:35:35Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" routerName=qs-backend-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/queue/`)" providerName=default.acme
time="2021-02-08T13:35:36Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" providerName=default.acme rule="Host(`ourdomain.example.com`) && PathPrefix(`/server-health`)" routerName=server-health-https@file
time="2021-02-08T13:35:39Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" routerName=config-ui-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/setup`)" providerName=default.acme
time="2021-02-08T13:35:39Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" providerName=default.acme routerName=config-ui-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/setup`)"
time="2021-02-08T13:35:39Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" providerName=default.acme rule="Host(`ourdomain.example.com`) && PathPrefix(`/manual`)" routerName=ourapp-manual-enterprise-https@file
time="2021-02-08T13:35:41Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" providerName=default.acme routerName=ws-backend-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/`)"
time="2021-02-08T13:35:42Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" providerName=default.acme routerName=qspage-frontend-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/queue-ui/`)"
time="2021-02-08T13:35:42Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" providerName=default.acme routerName=server-health-https@file rule="Host(`ourdomain.example.com`) && PathPrefix(`/server-health`)"
time="2021-02-08T13:35:43Z" level=debug msg="Certificates obtained for domains [ourdomain.example.com]" providerName=default.acme rule="Host(`ourdomain.example.com`) && PathPrefix(`/dex/`)" routerName=dex-https@file

@akhenakh @BernhardGruen @alesnav based on my experiments (see previous comment), this might be fixed in v2.4.
Can you check if you can still reproduce the bug in v2.4?

@heidemn-faro FWIW I'm seeing this in 2.4.8-1 running on Arch Linux with DigitalOcean as the DNS Provider and LE enabled.
It appears that Traefik is completely ignoring acme.json (it claims it's on a "read only filesystem" and I can assure you that it's not!) and therefore tries to regenerate the certs, causing the rate limiting to be triggered and I have to stop all TLS traffic for three days until the rate limit is lifted.
This is not running in a docker container, and I can read/write files as the Traefik user into the same directory that's storing acme.json so it's definitely a thing in Traefik.

Hey everyone, since there are a lot of fixes and tweaks applied in between 2.0.0 and the latest 2.9.1, including new options to configure the certificate duration lifecycle, I believe the original issue of a single Traefik instance retrying cert generation agressively is not a problem anymore.
Thus I propose to close this one in a few days if no new reports appear.


ignore case when lookup the service configurations with types

Expected Behavior
When lookup the service configurations based on types, the case shouldn't really matter. E.g. I can use "keycloak" or "Keycloak"
Actual Behavior
It does matter.

Looks like this is resolved.


feature(branchPrefix): Added git branch prefixing script.

Here's something that I've found quite useful. Let me know what you guys think!
When I develop new branches, I often forget their names. With this script, as long as you know the current month, you can start typing it, and then use tabbed autocomplete to get the rest of the way there.
git-new-branch myNewBranch  ---> 10202016-myNewBranch
To find it again, tab complete on the month October by typing 10

I think that's pretty neat! I don't think I'm going to add it straight to my dotfiles, though, purely just because I tend not to do date-based branches (I usually use puns in branch names, tbh). In any case, it should be easy enough for others to pull into their forks if they're interested, though! Appreciate the pull.


Remove node 12 support from CI

What does this PR do?
Removes node 12 support from CI.
Motivation
Node 12 is not longer supported in tracer.
Checklist

[ ] Unit tests have been updated and pass

With this PR, next release should be a major release.


Add .nav-underline modifier class

Exploring an idea here for adding a .nav-underline variant. Styles are pretty simple, but maybe it's still worth it?

Preview: https://deploy-preview-33126--twbs-bootstrap.netlify.app/docs/5.0/components/navs-tabs/#underline

Would it make sense to show the underline on hover/focus too? Also, how about using a transition from left to right using ::before and transform?
Just spitballing here :P

I'd go with a pseudo-element too, because it'd allow more fine tuning (position, background, playing with other shapes than line, etc.). BTW I'd consider this should be a basis and not a modifier since active state should not be conveyed with color only (which is the case as of now).
Boosted uses a pseudo-element to drax a line (stuck to the content-box) in .nav .active, thus applies it to every possible nav.
My two cents :D

Refreshed the design and added hover/focus styles. Not sure about building with a pseudo-element @ffoodd. Feels like that might be a little over-engineered for our default styles?

Probably, we'll see whatever issue might come up in the future.
The real ones IMHO regarding borders is altered box sizing, and not sticking to text content if we add horizontal padding; but this does not concern our base styles.
So ship it ☺


Limit(0, 1000) doesn't work

	// only serialize something if it's different than the default
	// The default is 0 10
	if p.Offset != DefaultOffset && p.Num != DefaultNum {
		args = args.Add("LIMIT", p.Offset, p.Num)
	}

Passing 0 in Offset causes to avoid passing LIMIT clause

hi there @Felixls you're totally right. #27 should fix it :)

oh sweet, thanks


salesforce-api.postman_export.json download

How do I download salesforce-api.postman_export.json  file to import it into Postman

Hi @cg-madhu. We no longer rely on the JSON file to install the collection, we're now using Postman Public Workspaces (more convenient, allows updates).
Take a look at the readme to see how you can install the collection with the Postman desktop app or web UI.


Update amazon-ecs-agent to expose 51679

Maps the ecs-agent credentials metadata port 51679 back to the ecs-agent container
This change is required in order to enable task iam roles as per : http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html#enable_task_iam_roles

cherry-picked into v1.1.0 and v1.0.3

Thank you kindly !


macOS quit shortcut is unexpected

Prerequisites
If the following boxes are not ALL checked, your issue is likely to be closed.

[x] Using yarn
[x] Using node 10.x
[x] Using an up-to-date master branch
[x] Using latest version of devtools. See wiki for howto update
[x] For issue in production release, devtools output of DEBUG_PROD=true yarn build && yarn start
[x] Tried solutions mentioned in #400

Describe the bug
On macOS, the app does not quit when using CMD+Q which is the "Quit" shortcut on macOS.
Expected behaviour
The app uses the macOS shortcut: CMD+Q.
Current Behavior
The app currently requires ALT+CMD+Q, which is an unusual and unexpected shortcut for macOS users.
Steps to Reproduce (for bugs)

Launch the app
Press CMD+Q
App does not quit

Your Environment
Include as many relevant details about the environment you experienced the bug in

macOS 10.15.4
OpenMTP 2.4.1 (downloaded from release page)

Screenshots

Replied: https://github.com/ganeshrvel/openmtp/pull/142#issuecomment-634633194

Fixed the issue.
https://github.com/ganeshrvel/openmtp/releases/tag/v2.4.4

Thank you!


Add params example

cherry-picked from https://discourse.gohugo.io/t/how-to-override-css-classes-with-hugo/3033/3

many thanks!


use another piece of hardware to record the track

As a fisherman I would like to be able to use another piece of hardware to record the track because that will not impact my phones power usage. https://github.com/StAResComp/sifids/issues/6, https://github.com/StAResComp/sifids/issues/8

could


[Feature Request] Ability to let providers specify the default active target.

Currently the default active target is set to be the first one in the target list.
What I need is to have my most used target (not the first one) set each time open the project.
I currently consider:

Add a active filed to target. If set to true it will be one of the default active target candidates.
A provider should only set one target to be active, If specify more than one the first with active set will be selected.
If multiple providers provides active target, the first one in the final targets list will be selected.
If no targets with active set to true, the first target in the final targets list will be selected.

This should be simple to add. I'll add a PR later.

I don't see how this is different from what exists today? The first target will be the prioritised one.
If multiple providers would specify the active target (and I bet every provider believes their targets are most important) it would still be "random" which target is the default one.
If you want a chance at being first, return it as the first target. Otherwise the user need to select by select-active-target.

I want my targets sort alphabetically, rather than by priorities.
eg:

Ant what I want to be selected is Mac: Run without build.
This does not change any thing for multiple providers.
But does provide more choice to single provider to have a way to make default target not in the first postion of list. And, for me, single provider is the most case.

I don't think it makes much sense. For a build provider developer there's no telling whether or not you will be the only build provider, hence there's no telling if the active setting  will have any affect at all.


Race condition in issue-credential v1.0 leads to credential exchange switching to abandoned state

We have a test system running that connects two ACA-Py instances and triggers issuance of credentials using the issue-credential v1.0 protocol. During load tests we have around 1% of all credential exchanges fail.
Some more details:

We listen for the /issue_credential webhook and wait for the status REQUEST_RECEIVED of the credential exchange
If we receive this status we invoke /issue-credential/records/{credentialExchangeId}/issue to trigger the issuance
This request sometimes fails with a 400 response and the error message reads "Credential exchange  in offer_sent state (must be request_received)." although our request is the direct result of a REQUEST_RECEIVED status event.
Afterwards the webhook returns with an error and is probably invoked again. This time the error involves the credential exchange state abandoned.

We found out that this problem can be mitigated by waiting some seconds before doing the issue http call and assumed thus, that this is a race condition. Further analysis seems to confirm this:

In https://github.com/hyperledger/aries-cloudagent-python/blob/aaee62f197f982d3b8d476e9ace690992708076f/aries_cloudagent/protocols/issue_credential/v1_0/manager.py#L560-L562 the credential record is saved and the the transaction is committed
Notification through the webhook happens inside of save in the post_save method in an asynchronous call inside of https://github.com/hyperledger/aries-cloudagent-python/blob/aaee62f197f982d3b8d476e9ace690992708076f/aries_cloudagent/transport/outbound/manager.py#L312. This way the transaction can commit before the webhook is invoked and thus the webhook can see the updated status.
We assume that most of the time we do the http call to trigger the issuance the transaction is already committed but sometimes this did not already happen and leads to the observed problem
When receiving the callback the transaction should be already committed to prevent race conditions.
This might also be a problem for other webhooks that involve transactions on the updated entities
The presented "workaround" of waiting is not a solution but more an approach to locate the problem. It prolongs the issuance by a large amount of time and does not work in all cases. The system should be race condition free to have something predictable.

FYI - @ianco -- this sounds familiar.

@swcurran  ... Yes, similar to the mediator issue we uncovered ...  we need to look at where we're sending messages vs committing updates to the exchange records, and also what error checking we're doing on the state when we receive a message (or invoke one of the admin api endpoints) ...

OK not exactly like the mediator issue, but also looks like a general problem across basically all protocols.
In the base_record (https://github.com/hyperledger/aries-cloudagent-python/blob/main/aries_cloudagent/messaging/models/base_record.py#L389) - the emit_event() happens before the transaction is committed (which I think is what eventually triggers the webhook).
Ideally any events that are triggered within a protocol shouldn't actually be "emitted" and acted upon until the protocol step completes.
Also the specific error of "exchange is in wrong state" shouldn't trigger the exchange to be pushed into an abandoned state or issue a problem report.

Labelling this as High Priority because I think this needs to be addressed for a 1.0.0 release

Need to relook at this issue.

@ianco -- could you look at this again, please?  As issue-credential-1.0 is deprecated, an issue local to that particular concern is not that big a deal. However, if it is a broader issue, we need to know that, and to characterize both the problem and what actions we should take.  Not looking for it to be fixed (yet) -- just a definition of the problem, its potential impact and suggestions.
Note that I didn't read through this full issue, so the information might already be all here. If so, summarizing it in one comment, or in an ACA-Pug presentation might be sufficient for this request.  Let me know.

I'll take a look

However, if it is a broader issue, we need to know that ...

I believe this is a bigger problem.  The emit() function (which triggers the webhook) happens as each record is added, updated or deleted, so in all cases the webhook will be called before the record is saved.  So we have potential race conditions pretty much everywhere.
My suggestion ... the problem is in the following code, in https://github.com/hyperledger/aries-cloudagent-python/blob/main/aries_cloudagent/messaging/models/base_record.py:
    async def post_save(
        self,
        session: ProfileSession,
        new_record: bool,
        last_state: Optional[str],
        event: bool = None,
    ):
        """Perform post-save actions.

        Args:
            session: The profile session to use
            new_record: Flag indicating if the record was just created
            last_state: The previous state value
            event: Flag to override whether the event is sent
        """

        if event is None:
            event = new_record or (last_state != self.state)
        if event:
            await self.emit_event(session, self.serialize())

    async def delete_record(self, session: ProfileSession):
        """Remove the stored record.

        Args:
            session: The profile session to use
        """

        if self._id:
            storage = session.inject(BaseStorage)
            if self.state:
                self._previous_state = self.state
                self.state = BaseRecord.STATE_DELETED
                await self.emit_event(session, self.serialize())
            await storage.delete_record(self.storage_record)

    async def emit_event(self, session: ProfileSession, payload: Any = None):
        """Emit an event.

        Args:
            session: The profile session to use
            payload: The event payload
        """

        if not self.RECORD_TOPIC:
            return

        if self.state:
            topic = f"{self.EVENT_NAMESPACE}::{self.RECORD_TOPIC}::{self.state}"
        else:
            topic = f"{self.EVENT_NAMESPACE}::{self.RECORD_TOPIC}"

        if not payload:
            payload = self.serialize()

        await session.profile.notify(topic, payload)

(There are other scenarios as well in this code.)
Note that in the delete() scenario the emit() is called before the record is saved, however in the post_save() scenario it happens before.  In either case, the commit() happens later on.
My suggestion is to move the emit() code into the commit() method in the SessionProfile class:  https://github.com/hyperledger/aries-cloudagent-python/blob/main/aries_cloudagent/core/profile.py
This would ensure that all notifications happen after the transaction is committed (and database updates are updated) however it may have other side effects.  I can give this a try but wanted to get some feedback ...
@dbluhm   @swcurran   @shaangill025   any thoughts?

My suggestion is to move the emit() code into the commit() method in the SessionProfile class: https://github.com/hyperledger/aries-cloudagent-python/blob/main/aries_cloudagent/core/profile.py
This would ensure that all notifications happen after the transaction is committed (and database updates are updated) however it may have other side effects. I can give this a try but wanted to get some feedback ...

the emit_event() method in the SessionProfile class would just cache the event in a local array
when ommit() is called, it would emit all the events (after the database commit)
in the case of rollback() all events would be abandoned

@ianco - could you prepare a short session on this for the ACA-Pug meeting tomorrow.  I’d like this to have a higher profile, since you haven’t gotten feedback.  This is out of my realm.

@ianco - could you prepare a short session on this for the ACA-Pug meeting tomorrow. I’d like this to have a higher profile, since you haven’t gotten feedback. This is out of my realm.

Yep will do.  I have a fix so I'll open a PR.  I'm having trouble duplicating the issue though so can't (yet) verify the fix.

Fixed by #2760

@markuskreusch


og:image:alt support

I've discovered Facebook supporting a og:image:alt parameter. Do you think its worth to add in? I could do it, just checking beforehand

Hi Peter,
That's a good idea. Do you want to submit a PR?
Cheers

Sure, I'll prepare one shortly :+1:

Oh, I forgot about this. Sorry about that :/
On Fri, 20 Sep 2019 07:44 Florian Thoma<EMAIL_ADDRESS>wrote:

Closed #4 https://github.com/xini/silverstripe-social-metadata/issues/4
via 2cb67df
https://github.com/xini/silverstripe-social-metadata/commit/2cb67df8f1fa2229e3f350f1f0f2a9a66d414324
.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/xini/silverstripe-social-metadata/issues/4?email_source=notifications&email_token=ACAK7MYLCVBTY5LB4JL6R53QKR5PXA5CNFSM4HY37PU2YY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOTXVPCZI#event-2649420133,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACAK7M2Q6DUKO2OJN6RDQ2DQKR5PXANCNFSM4HY37PUQ
.

No worries ;)


Use anonymous functions instead of create_function()

create_function() is deprecated in PHP v7.2.0

Thanks for this! Please could you convert the square bracket definitions back to array() declarations? Anonymous functions are supported for php 5.3+ but square brackets don't come in until 5.4+. In theory WordPress plugins ought to work back to 5.2.4 (I know, crazy--but a lot of site still run really old versions of php). In the case where 5.2.4 compatibility breaks 7.x functionality (i.e. anonymous versus create_function functions) I'm siding with 7.x of course. But if we can keep it back-compat. without extra hassle, let's do it.
Looking forward to the revised pull req. Thanks again!

Fixed! I realized that I introduced a some usages of the short array syntax in previous commits, so I've opened a PR to fix those as well.

Awesome, thanks!


How do I visualize Convnet layer weights

Hi,
I am interested in visualizing the filters of my convnet.
Here is my model:
model = Sequential()

model.add(Convolution2D(32, 5, 5, border_mode='same',name='conv1_1', input_shape = (1, 106, 106)))

convout1 = model.layers[1]

model.add(Activation("relu"))
model.add(Convolution2D(32, 5, 5, border_mode='same',name='conv1_2'))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

#2
model.add(Convolution2D(64, 5, 5, border_mode='same',name='conv2_1'))
model.add(Activation("relu"))
model.add(Convolution2D(64, 5, 5, border_mode='same',name='conv2_2'))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

#flatten
model.add(Flatten())
model.add(Dropout(0.5))

model.add(Dense(2))
model.add(Activation('softmax'))

rms = RMSprop()
sgd = SGD(lr=0.001, decay=1e-6, momentum=0.5, nesterov=True)
model.compile(loss='categorical_crossentropy', optimizer=sgd,metrics=["accuracy"])

I tried doing something like this (as found here):
convout1_f = theano.function([model.get_input(train=False)], convout1.get_output(train=False))
convolutions = convout1_f(reshaped[img_to_visualize: img_to_visualize+1])

But get an error:

AttributeError: 'Sequential' object has no attribute 'get_input'

Any help would be appreciated.

[X ] Check that you are up-to-date with the master branch of Keras. You can update with:
pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps
[ X] If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps

try this
theano.function([model.input], convout1)
also you might want to do
from keras import backend as K
and then
K.function([model.input], convout1)

@AdityaGudimella, Thanks for the quick reply, I get:
TypeError: Outputs must be theano Variable or Out instances. Received <keras.layers.core.Activation object at 0x7f2045298190> of type <class 'keras.layers.core.Activation'>

It is a little unclear but I tried this (found here):
################
# the name of the layer we want to visualize (see model definition below)
layer_name = 'conv2_2'

# util function to convert a tensor into a valid image
def deprocess_image(x):
    # normalize tensor: center on 0., ensure std is 0.1
    x -= x.mean()
    x /= (x.std() + 1e-5)
    x *= 0.1

    # clip to [0, 1]
    x += 0.5
    x = np.clip(x, 0, 1)

    # convert to RGB array
    x *= 255
    x = x.transpose((1, 2, 0))
    x = np.clip(x, 0, 255).astype('uint8')
    return x

layer_dict = dict([(layer.name, layer) for layer in model.layers])

def normalize(x):
    # utility function to normalize a tensor by its L2 norm
    return x / (K.sqrt(K.mean(K.square(x))) + 1e-5)

kept_filters = []
for filter_index in range(0, 64):
    # we only scan through the first 200 filters,
    # but there are actually 512 of them
    print('Processing filter %d' % filter_index)
    start_time = time.time()

    # we build a loss function that maximizes the activation
    # of the nth filter of the layer considered
    layer_output = layer_dict[layer_name].output
    loss = K.mean(layer_output[:, filter_index, :, :])

    # we compute the gradient of the input picture wrt this loss
    grads = K.gradients(loss, input_img)[0]

    # normalization trick: we normalize the gradient
    grads = normalize(grads)

    # this function returns the loss and grads given the input picture
    iterate = K.function([input_img, K.learning_phase()], [loss, grads])

    # step size for gradient ascent
    step = 1.

    # we start from a gray image with some random noise
    input_img_data = np.random.random((1, 1, 106, 106)) * 20 + 128.

    # we run gradient ascent for 20 steps
    for i in range(20):
        loss_value, grads_value = iterate([input_img_data,0])
        input_img_data += grads_value * step

        print('Current loss value:', loss_value)
        if loss_value <= 0.:
            # some filters get stuck to 0, we can skip them
            break

    # decode the resulting input image

    img = deprocess_image(input_img_data[0])
    kept_filters.append((img, loss_value))
#    if loss_value > 0:
#        img = deprocess_image(input_img_data[0])
#        kept_filters.append((img, loss_value))
    end_time = time.time()
    print('Filter %d processed in %ds' % (filter_index, end_time - start_time))

# we will stich the best 64 filters on a 8 x 8 grid.
n = 3

# the filters that have the highest loss are assumed to be better-looking.
# we will only keep the top 64 filters.
kept_filters.sort(key=lambda x: x[1], reverse=True)
kept_filters2 = kept_filters[:n * n]

# build a black picture with enough space for
# our 8 x 8 filters of size 128 x 128, with a 5px margin in between
margin = 3
width = n * img_width + (n - 1) * margin
height = n * img_height + (n - 1) * margin
stitched_filters = np.zeros((width, height, 3))

# fill the picture with our saved filters
for i in range(n):
    for j in range(n):
        img, loss = kept_filters2[i * n + j]
        stitched_filters[(img_width + margin) * i: (img_width + margin) * i + img_width,
                         (img_height + margin) * j: (img_height + margin) * j + img_height, :] = img

# save the result to disk
imsave('stitched_filters_%dx%d.png' % (n, n), stitched_filters)

I got the same error traceback...

Hi,
I think I was working off the same link you were:
https://github.com/julienr/ipynb_playground/blob/master/keras/convmnist/keras_cnn_mnist.ipynb
But I had to change a bunch of stuff.  Here's what works for me:
MNISTVisualizeOutputsExample.zip

I'm getting the gradient as None always.
print(loss)
>>> Tensor("Mean_4:0", shape=(), dtype=float32)
print(input_img)
>>>array([[[[104, 108, 102, ...,  12,  17,  18],
         [ 18,  23,  22, ...,  61,  65,  59],
         [ 70,  71,  62, ...,   6,   4,   3],
         ...,
print(grads = K.gradients(loss, input_img)[0])
>>>None

I get the same error
AttributeError: 'Sequential' object has no attribute 'get_input'

try this below
theano.function([model.input], convout1)

and get another error
TypeError: Outputs must be theano Variable or Out instances. Received <keras.layers.core.Activation object at 0x000000606F9BC1D0> of type <class 'keras.layers.core.Activation'>


ARROW-8656: [Python] Switch to VS2017 in the windows wheel builds

@github-actions crossbow submit wheel-win-*

@github-actions crossbow submit wheel-win-cp38

@github-actions crossbow submit wheel-win-cp38

@github-actions crossbow submit wheel-win-cp38

@github-actions crossbow submit wheel-win-cp38

@github-actions crossbow submit wheel-win-cp38

@github-actions crossbow submit wheel-win-*

@github-actions crossbow submit wheel-win-*

@xhochy to run the wheels, or build them?

@xhochy to run the wheels, or build them?

To run.

Is this change necessary? I understand why we are using VS2017 in the conda package but why in the wheels?

We would need to switch away from using the compiled conda packages as the basis for the wheels. I'm not sure if there is a different source that could provide except Arrow building them on their own.
With my Arrow hat on, this is really annoying as either build times will increase significantly or we would have to use make a pre-filled Windows Docker image that already includes all our static libraries.
With my conda-forge hat on, I would actually love to see this happen as this would mean that we could get rid of the static libraries which we actually don't want to build and distribute. As a package manager who can explicitly deal with native dependencies, dynamic linkage makes the package maintenance a whole lot easier.

Actually, I think the most practical approach would be to use the conda-forge recipes, patch them to use vs2015 and upload them to a separate channel. This gives us the build machinery to not rebuild them in every CI run and also keeps us from maintaining a complete set of build scripts, we just need to copy them over from time to time.

We could also use BUNDLED builds for the dependencies of interest (might also be nice to turn on ARROW_MIMALLOC while we are at it). We'd need to turn off Gandiva, though

I'm not sure what's the verdict here, but this issue becomes pressing because of a possible upcoming patch release.
The python 3.5 script uses bundled dependencies except for boost python and zlib (no gandiva, no flight). I can try to switch back to vs2015 if you would prefer that.

Yeah, IMHO let's change to bundled dependencies and stick with VS2015 until we understand the implications of moving to the newer runtime.

Closing in favor of https://github.com/apache/arrow/pull/7129


Customizable Border Type

Would be good if we can customize the border type when its focused or not.
will be needing this to match the rounded borders on my Textual program

Textual now supports inputs natively -> https://textual.textualize.io/widgets/input/
Textual support of CSS -> https://textual.textualize.io/css_types/


Add ToCsv method with test

Seems OK as an initial version but what will it do if the strings in the enumerable themselves have , in them? See https://stackoverflow.com/a/4617967/13729 for example. could be a follow-up enhancement.

Yeah you are right it needs more enhancement I'II work on it.


FlywayException: No database found to handle jdbc:mysql://localhost:3306/...

Which version and edition of Flyway are you using?
8.5.11
If this is not the latest version, can you reproduce the issue with the latest one as well? (Many bugs are fixed in newer releases and upgrading will often resolve the issue)
Latest
Which client are you using? (Command-line, Java API, Maven plugin, Gradle plugin)
Java API
Which database are you using? (Type & version)
MySQL 8.0.25
Which operating system are you using?
Windows / Linux WSL
What did you do? (Please include the content causing the issue, any relevant configuration settings, the SQL statement(s) that failed (if any), and the command you ran)
Upgraded Flyway from 7.14.0 to 8.5.11. Ran the code again, got an exception on migration:
org.flywaydb.core.api.FlywayException: No database found to handle jdbc:mysql://localhost:3306/mydb?useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC
    at org.flywaydb.core.internal.database.DatabaseTypeRegister.getDatabaseTypeForUrl(DatabaseTypeRegister.java:55)
    at org.flywaydb.core.internal.jdbc.DriverDataSource.<init>(DriverDataSource.java:109)
    at org.flywaydb.core.internal.jdbc.DriverDataSource.<init>(DriverDataSource.java:80)
    at org.flywaydb.core.api.configuration.ClassicConfiguration.setDataSource(ClassicConfiguration.java:958)
    at org.flywaydb.core.api.configuration.FluentConfiguration.dataSource(FluentConfiguration.java:683)
    at common.db.RelationalDB.migrateRelationalDB(RelationalDB.java:131)
    at app.Main.newServer(Main.java:59)
    at app.Main.lambda$main$0(Main.java:50)
    at common.util.AppInitializer.buildApp(AppInitializer.java:21)
    at app.Main.main(Main.java:30)

The migrateRelationalDB() function calls Flyway this way:
Flyway.configure().initSql(init).locations(location).dataSource(url, user, pass).load().migrate();

I managed to get the code to work by downgrading to 8.1.0
The app is a fatjar that includes mysql:mysql-connector-java:8.0.29.
What did you expect to see?
No error. Normal connection and working app.
What did you see instead?
Can't start up if the migration doesn't succeed. (There are no new migrations, Flyway only needs to verify that everything is still up to date)

See the MySQL docs here
You'll need a dependency on the flyway-mysql module

I had the library but it was still failing. Turns out my build is rearranging the dependencies in alphabetical order which made the core library always come first. Which meant that my packaging step always included the META-INF/services/org.flywaydb.core.extensibility.Plugin from the core library instead of the mysql library. And that made it fail. I have some work to do.

Thanks almost 2 hours with this problem I didn't think in the versions.


Anomaly Heatmaps

Hi,
is there a way to get the anomaly heatmaps from your code?
Best regards,
Justus

padim.predict(..) returns the mahalanobis distances for each patch of the image. You can resize it to a square matrix to see the unormalized anomaly map. So you can do something like that:
import math
import matplotlib.pyplot
distances = padim.predict(img.unsqueeze(0))
w = math.sqrt(distances.numel())
amap = distances.view(w, w)
plt.imshow(amap)
plt.show()

Here is an example notebook where the input is 416x416 so the backbone outputs 104x104 anomaly maps

Thank you again for your super fast work. I appreciate your effort very much!

@Pangoraw Why are we using Gaussian blurring as a preprocessing step for the amap? Is it just for visualizing or is there any specific reason for doing that?