Pushing repository fails

I'm trying to push a fairly large git repository for the first time. It fails every time I try.
Error message included below.

rk $ git push -v
Pushing to dropbox:///Repos/research1
debug: writing: /repos/research1/objects/1a/6eda2e816781aa27e8b09ca99553c64cca5846
[... deleted lines ...]
debug: writing: /repos/research1/objects/33/7ea6e0aa8efe47a0b396760871e44f6ba8968a
Traceback (most recent call last):
File "/anaconda/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/anaconda/lib/python3.6/site-packages/urllib3/util/connection.py", line 60, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/anaconda/lib/python3.6/socket.py", line 743, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/anaconda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/anaconda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "/anaconda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
conn.connect()
File "/anaconda/lib/python3.6/site-packages/urllib3/connection.py", line 284, in connect
conn = self._new_conn()
File "/anaconda/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x11171b860>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/anaconda/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
timeout=timeout
File "/anaconda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/anaconda/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='content.dropboxapi.com', port=443): Max retries exceeded with url: /2/files/upload (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x11171b860>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/anaconda/bin/git-remote-dropbox", line 11, in
sys.exit(main())
File "/anaconda/lib/python3.6/site-packages/git_remote_dropbox/init.py", line 759, in main
helper.run()
File "/anaconda/lib/python3.6/site-packages/git_remote_dropbox/init.py", line 372, in run
self._do_push(line)
File "/anaconda/lib/python3.6/site-packages/git_remote_dropbox/init.py", line 409, in _do_push
self._push(src, dst)
File "/anaconda/lib/python3.6/site-packages/git_remote_dropbox/init.py", line 461, in _push
for done, _ in enumerate(res, 1):
File "/anaconda/lib/python3.6/multiprocessing/pool.py", line 699, in next
raise value
File "/anaconda/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/anaconda/lib/python3.6/site-packages/git_remote_dropbox/init.py", line 295, in call
return method(self._obj, *args)
File "/anaconda/lib/python3.6/site-packages/git_remote_dropbox/init.py", line 524, in _put_object
self._connection().files_upload(data, path, mode, mute=True)
File "/anaconda/lib/python3.6/site-packages/dropbox/base.py", line 1267, in files_upload
f,
File "/anaconda/lib/python3.6/site-packages/dropbox/dropbox.py", line 234, in request
timeout=timeout)
File "/anaconda/lib/python3.6/site-packages/dropbox/dropbox.py", line 325, in request_json_string_with_retry
timeout=timeout)
File "/anaconda/lib/python3.6/site-packages/dropbox/dropbox.py", line 409, in request_json_string
timeout=timeout,
File "/anaconda/lib/python3.6/site-packages/requests/sessions.py", line 555, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/anaconda/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/anaconda/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/anaconda/lib/python3.6/site-packages/requests/adapters.py", line 508, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='content.dropboxapi.com', port=443): Max retries exceeded with url: /2/files/upload (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x11171b860>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))
error: failed to push some refs to 'dropbox:///Repos/research1'

How large is the git repository? Can you give me instructions on how to reproduce this?
I tried two things. I tried a large git repository with lots of small files and many medium-sized files (about 1000 objects total, and 100M total), and it worked for me.
I also tried the following:
cd /tmp
mkdir test
cd test
# make a bunch of random large files
dd if=/dev/urandom of=10m.bin bs=1M count=10
dd if=/dev/urandom of=25m.bin bs=1M count=25
dd if=/dev/urandom of=50m.bin bs=1M count=50
dd if=/dev/urandom of=100m.bin bs=1M count=100
git init
git add .
git commit -m 'bla'
git push dropbox:///test master:master

This also worked just fine for me.

Any updates?

Hi,
Your code also worked for me, but going for slightly larger files caused problems.
The following produced an error for me:
mkdir test2
cd test2

dd if=/dev/urandom of=10m.bin bs=1m count=10
dd if=/dev/urandom of=25m.bin bs=1m count=25
dd if=/dev/urandom of=50m.bin bs=1m count=50
dd if=/dev/urandom of=100m.bin bs=1m count=200

git init
git add .
git commit -m 'bla'
git push -v dropbox:///repos/test2 master:master

Output was
git push -v dropbox:///repos/test2 master:master
Pushing to dropbox:///repos/test2
debug: writing: /repos/test2/objects/5f/8e4b42782f7371960f07221309852cc73ab3c9
debug: writing: /repos/test2/objects/df/d3a33b02c59fbaa8a58737cfe3e34cab2f0d76
debug: writing: /repos/test2/objects/5e/462ee0e672e3db01856ce741d8948237894328
debug: writing: /repos/test2/objects/f9/3e93168fe093e67561873e0037bc50ac242e42
debug: writing: /repos/test2/objects/6d/98c751aa01c10121d73025a754739229ff33e0
debug: writing: /repos/test2/objects/c8/9dabfd26f91ed832b8457bcd8e1c81ebca3a12

Traceback (most recent call last):
  File "/anaconda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/anaconda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 383, in _make_request
    httplib_response = conn.getresponse()
  File "/anaconda/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/anaconda/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/anaconda/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/anaconda/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "/anaconda/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 294, in recv_into
    raise timeout('The read operation timed out')
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/anaconda/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/anaconda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/anaconda/lib/python3.6/site-packages/urllib3/util/retry.py", line 357, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/anaconda/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
    raise value
  File "/anaconda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/anaconda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 389, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/anaconda/lib/python3.6/site-packages/urllib3/connectionpool.py", line 309, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='content.dropboxapi.com', port=443): Read timed out. (read timeout=30)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/anaconda/bin/git-remote-dropbox", line 11, in <module>
    sys.exit(main())
  File "/anaconda/lib/python3.6/site-packages/git_remote_dropbox/__init__.py", line 759, in main
    helper.run()
  File "/anaconda/lib/python3.6/site-packages/git_remote_dropbox/__init__.py", line 372, in run
    self._do_push(line)
  File "/anaconda/lib/python3.6/site-packages/git_remote_dropbox/__init__.py", line 409, in _do_push
    self._push(src, dst)
  File "/anaconda/lib/python3.6/site-packages/git_remote_dropbox/__init__.py", line 461, in _push
    for done, _ in enumerate(res, 1):
  File "/anaconda/lib/python3.6/multiprocessing/pool.py", line 699, in next
    raise value
  File "/anaconda/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/anaconda/lib/python3.6/site-packages/git_remote_dropbox/__init__.py", line 295, in __call__
    return method(self._obj, *args)
  File "/anaconda/lib/python3.6/site-packages/git_remote_dropbox/__init__.py", line 524, in _put_object
    self._connection().files_upload(data, path, mode, mute=True)
  File "/anaconda/lib/python3.6/site-packages/dropbox/base.py", line 1267, in files_upload
    f,
  File "/anaconda/lib/python3.6/site-packages/dropbox/dropbox.py", line 234, in request
    timeout=timeout)
  File "/anaconda/lib/python3.6/site-packages/dropbox/dropbox.py", line 325, in request_json_string_with_retry
    timeout=timeout)
  File "/anaconda/lib/python3.6/site-packages/dropbox/dropbox.py", line 409, in request_json_string
    timeout=timeout,
  File "/anaconda/lib/python3.6/site-packages/requests/sessions.py", line 555, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/anaconda/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/anaconda/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/anaconda/lib/python3.6/site-packages/requests/adapters.py", line 521, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='content.dropboxapi.com', port=443): Read timed out. (read timeout=30)
error: failed to push some refs to 'dropbox:///repos/test2'

Huh. Well, I don't think we can do anything about this -- this isn't a bug in git-remote-dropbox, I think it's an issue with the Dropbox API. I'm not sure we can work around it without complicating our on-disk format (by e.g. splitting up large files).

The doc clearly states that files_upload should not be used with files greater than 150MB and instead use an upload session with files_upload_session_start.
I'll be happy to implement it, when I have the time.

Oh haha that's a good catch!
Sure, it'll be great to have this feature. (Though I guess it isn't super critical since most people probably won't have > 150MB files in their repo)

I've created PR #75 which I believe addresses this issue.


Ability to define task dependencies with rshift and lshift (e.g. like in Airflow DAGs)

Expected Behavior
Ability to define task dependencies with rshift and lshift (e.g. like in Airflow DAGs)
Current Behavior
NA
Steps to Reproduce (for bugs)
Context
Your Environment

pyjaws version used:

Solved with https://github.com/rafaelpierre/pyjaws/tree/feature/shift-operators


Spanner: Use gax for retries

Use GAX instead of the custom SpannerImpl.runWithRetries method for retrying server calls.
This migration involves three changes:

Remove the override of the generated default settings in GapicSpannerRpc that set the retryable error codes to an empty set. This will automatically enable GAX retries for all server calls using the default values that have already been defined. The defaults are the same as the defaults that are used by SpannerClient.
Add configuration options to SpannerOptions to allow the user to change the default RetrySettings. As SpannerOptions is used to create a Spanner instance, which again contains both a DatabaseClient, DatabaseAdminClient and InstanceAdminClient, SpannerOptions must contain configuration options for all three underlying clients.
The runWithRetries method in SpannerImpl is removed and all references to it changed into direct calls that will automatically be retried by GAX.

To avoid a lot of unnecessary configuration options on SpannerOptions, the retry settings of the three underlying clients are exposed by exposing the StubSettings.Builders that are used to create the underlying clients. This only adds three extra methods to SpannerOptions, while still granting (advanced) users access to all configuration options.
The client library will use the defaults that are globally defined for retries in the configuration yaml files of Spanner (i.e. in the files spanner_gapic.yaml, spanner_admin_database_gapic.yaml and spanner_admin_instance_gapic.yaml)
The following defaults are defined:

Spanner default settings:

Initial retry delay: 1 second
Retry multiplier: 1.3
Max retry delay: 32 seconds
Initial RPC timeout: 60 seconds
RPC timeout multiplier: 1.0
Max RPC timeout: 60 seconds
Total timeout: 600 seconds

Spanner streaming settings (used for executeStreamingSql and streamingRead):

Initial retry delay: 1 second
Retry multiplier: 1.3
Max retry delay: 32 seconds
Initial RPC timeout: 120 seconds
RPC timeout multiplier: 1.0
Max RPC timeout: 120 seconds
Total timeout: 1200 seconds

Spanner long-running settings (used for commit):

Initial retry delay: 1 second
Retry multiplier: 1.3
Max retry delay: 32 seconds
Initial RPC timeout: 3600 seconds
RPC timeout multiplier: 1.0
Max RPC timeout: 3600 seconds
Total timeout: 3600 seconds

Database admin default settings (used for all methods except methods that return an operation, i.e. createDatabase and updateDatabaseDdl, these have a total timeout of 24 hours):

Initial retry delay: 1 second
Retry multiplier: 1.3
Max retry delay: 32 seconds
Initial RPC timeout: 60 seconds
RPC timeout multiplier: 1.0
Max RPC timeout: 60 seconds
Total timeout: 600 seconds

Instance admin default settings (used for all methods except methods that return an operation, i.e. createInstance and updateInstance, these have a total timeout of 24 hours):

Initial retry delay: 1 second
Retry multiplier: 1.3
Max retry delay: 32 seconds
Initial RPC timeout: 60 seconds
RPC timeout multiplier: 1.0
Max RPC timeout: 60 seconds
Total timeout: 600 seconds

Codecov Report

Merging #4940 into master will increase coverage by 0.52%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #4940      +/-   ##
============================================
+ Coverage     49.98%   50.51%   +0.52%
- Complexity    23668    24049     +381
============================================
  Files          2238     2250      +12
  Lines        226007   228176    +2169
  Branches      24191    25126     +935
============================================
+ Hits         112974   115252    +2278
- Misses       103693   104272     +579
+ Partials       9340     8652     -688

Impacted Files
Coverage Δ
Complexity Δ

...re/src/main/java/com/google/cloud/BaseService.java
55.55% <0%> (-6.95%)
2% <0%> (ø)

...le/cloud/compute/deprecated/DeprecationStatus.java
88.63% <0%> (-3.13%)
20% <0%> (ø)

...able/admin/v2/stub/GrpcBigtableTableAdminStub.java
92.41% <0%> (-2.07%)
40% <0%> (+20%)

...e/admin/v2/stub/GrpcBigtableInstanceAdminStub.java
93.66% <0%> (-1.85%)
50% <0%> (+25%)

...va/com/google/cloud/http/HttpTransportOptions.java
56.33% <0%> (-1.64%)
10% <0%> (ø)

.../cloud/bigtable/data/v2/stub/GrpcBigtableStub.java
92.85% <0%> (-1.63%)
21% <0%> (+8%)

...table/admin/v2/BaseBigtableTableAdminSettings.java
10.9% <0%> (-1.16%)
4% <0%> (+2%)

...ain/java/com/google/cloud/logging/LoggingImpl.java
87.94% <0%> (-0.7%)
77% <0%> (+4%)

...le/admin/v2/BaseBigtableInstanceAdminSettings.java
8.63% <0%> (-0.58%)
4% <0%> (+2%)

.../google/cloud/bigquery/spi/v2/HttpBigQueryRpc.java
6.92% <0%> (-0.06%)
2% <0%> (ø)

... and 182 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2eaa9f4...28935c3. Read the comment docs.

This PR will also solve the original issue mentioned in #3616, as it will enable users to configure timeouts and custom retry settings for a SpannerOptions instance.


Blocking of the process until the player load is complete

Is your feature request related to a problem? Please describe.
Since it cannot detect when the player is finished loading, it cannot know when to terminate the program.
Describe the solution you'd like
Thread blocking process until the player finishes loading the music stream (like player.waitReady().get())
Additional context
Example
public class Main(){
    public static void main(String[] args) {
        Session session = new Session.Builder()
                            .userPass("UserName", "Password")
                            .create();
        PlayerConfiguration playerConfiguration = new PlayerConfiguration.Builder()
                    .setOutput(AudioOutput.STDOUT)
                    .build();
        Player player = new Player(playerConfiguration, session);
        try{
            player.waitReady().get();
        } catch(InterruptedException e){
            e.printStackTrace();
        } catch(ExecutionException e){
            e.printStackTrace();
        }
        player.load("spotify:track:xxxxxxxxxxxxxxxxxxxxxx", true);
// Something to block it?
        try{
            session.close();
        } catch(IOException e){
            e.printStackTrace();
        }
        System.exit(0);
    }
}

You can use Player#addEventsListener.
Side note:

We (the librespot-org organization and me) DO NOT encourage piracy and DO NOT support any form of downloader/recorder designed with the help of this repository and in general anything that goes against the Spotify ToS. If you're brave enough to put at risk this entire project, just don't publish it. This is meant to provide support for all those devices that are not officially supported by Spotify.

You can use Player#addEventsListener.
Side note:

We (the librespot-org organization and me) DO NOT encourage piracy and DO NOT support any form of downloader/recorder designed with the help of this repository and in general anything that goes against the Spotify ToS. If you're brave enough to put at risk this entire project, just don't publish it. This is meant to provide support for all those devices that are not officially supported by Spotify.

Player.Listener could not catch that the music data output was completed.
The stdout outputs the following events.
Player.EventsListener#onContextChanged spotify:track:xxxxxxxxxxxxxxxxxxxxxx
Player.EventsListener#onTrackChanged TrackId{spotify:track:xxxxxxxxxxxxxxxxxxxxxx},null
Player.EventsListener#onPlaybackResumed 18
Player.EventsListener#onMetadataAvailable xyz.gianlu.librespot.player.TrackOrEpisode@3ed0a6d6
{Music Data}
Player.EventsListener#onPlaybackHaltStateChanged true,839
{Music Data}
Player.EventsListener#onPlaybackHaltStateChanged false,839
{MusicData}
...

I'd like to use it to play Spotify on my Discord bot.

Side note:

We (the librespot-org organization and me) DO NOT encourage piracy and DO NOT support any form of downloader/recorder designed with the help of this repository and in general anything that goes against the Spotify ToS. If you're brave enough to put at risk this entire project, just don't publish it. This is meant to provide support for all those devices that are not officially supported by Spotify.

Player.Listener could not catch that the music data output was completed.
The stdout outputs the following events.
Player.EventsListener#onContextChanged spotify:track:xxxxxxxxxxxxxxxxxxxxxx
Player.EventsListener#onTrackChanged TrackId{spotify:track:xxxxxxxxxxxxxxxxxxxxxx},null
Player.EventsListener#onPlaybackResumed 18
Player.EventsListener#onMetadataAvailable xyz.gianlu.librespot.player.TrackOrEpisode@3ed0a6d6
{Music Data}
Player.EventsListener#onPlaybackHaltStateChanged true,839
{Music Data}
Player.EventsListener#onPlaybackHaltStateChanged false,839
{MusicData}
...

I'd like to use it to play Spotify on my Discord bot.

Side note:

We (the librespot-org organization and me) DO NOT encourage piracy and DO NOT support any form of downloader/recorder designed with the help of this repository and in general anything that goes against the Spotify ToS. If you're brave enough to put at risk this entire project, just don't publish it. This is meant to provide support for all those devices that are not officially supported by Spotify.

I have added the onPlaybackEndend event.

I have added the onPlaybackEndend event.

Thank you for making the change.

Thank you for making the change.


Augmentation of the fields fields transcribed into the initial csv file

The current set of fields transcribed by the transcriber were established some 15 years ago and are described in the SSCENS document. It does not include all of the fields submitted into the database by the current vld file.
As part of the on-line uploading process we need to establish how those fields are to be entered.
We also need to consider any additional fields that are desirable for the present suite of census documents.
We need further to consider what additional fields are required for the 1901 and 1911 census documents

@geoffj-FUG wrote the following: When should the Ecclesiastical Parish be added?
To do this review right I think that we should review when the Ecclesiastical Parish is added. Why? Because the answer affects CSVCheck.
Initially it was added by the proofreader in winCC. With the demise of WinCC it was added by the Validator (mostly by fudging!)
I believe that as it appears on every census page it should be added by t he transcriber and therefore allowed for in the transcription spreadsheet / csvfile.
In 1911 it moves to the fly sheet at the beginning of each ED.
So once recorded it just needs carrying down, not the tedious process used in winCC.
In 1911 This sheet also records Urban Boroughs and Parliamentary Boroughs.
So what and how when we record it needs to be thrown into the pot now, not later when Kirk is past the point of no return.

We also need to consider what is recorded in each field, given that the new system will not have the character length constraints of the current system.

It is important that we move systematically. The initial implementation has to function with current specification.

Brenda commented:
Just been giving this a little thought.
The EP is already in the PARMS CSV in column two.
Ie.
Column one has the piece number
Column 2 has the EP. (The single a in column H)
Column 3 has the Civil Parish (The single b, in column H)
Row one you will see Census year/county example. DOR1851
Etc.,
Why cannot this be picked up by the transcribe for FC2 (perhaps insert a new column  at the start of the spreadsheet so they are all at the start)
As they are on the PARMS they should be on the new FC2 spreadsheet on a list from the PARMS  for them to select along with the Civil Parish?
VALDREV also gives them an option to select a different CHP and POB.
This creates an extra column in the VLD CSV last stage.
Will this column still be needed as hopefully the facility to select a CHP and POB will be more advanced with the new way of selecting from a drop down list.
( well I am thinking/assuming, that is what will happen?)
So just a thought to add a column at the start of the CSV and remove a column at the end that might not be needed.
But then I have just got up and not at my best after a bad night, so will think it over again when I do wake up.  But it is a start of my thinking, so if any one thinks it might not work, then OK, just one of my thoughts.
The EP on the final VLD spreadsheet is the last column.
Why not make it the first from the dropdown list from the PARMS file.
I am sure I might have missed something so over to you all.

@FreecenBren @geoffj-FUG @PatReynolds
Since the online upload of a csv file this together with the Params should contain the total set of information required to create a search record now is the time to consider the augmentation of the fields in the csv file.
Rather than include a discussion on each augmentation in this story and end up losing sight of one I would like to suggest a separate story for each.
CEN is in a good position to add new fields in the csv file at it has always used a column name header as the second line in the file. Hence augmentations can be added. This what in REG was termed a flexible csv file. Provided the column header line is included we can be flexible in terms of their order and their number. We can accommodate the situation of absent column header line by assuming the file has the SSCENS layout.
Additional fields that need to be accommodated are:
Ecclesiastical Parish
1901 Census fields
1911 Census fields
Scottish windows field 1851 onwards
deleted flag

Kirk
My view is still that it should be added by the transcriber. It is closely related to the ED. The ED number occurs once in each ED and carries down. I would but it in the spreadsheet next to the ED number.
It is my belief that the EP can on rare occasions change within an ED. Therefore the logical place to put it in a spreadsheet is immediately to the right of the ED number. So the sequence becomes ED number, Ecclesiastical Parish, Folio etc.
I cannot agree with Brenda that the a record in the PARMS is the EP. The National Archives state that the a record is the Registration Sub-District and the b record is the Civil Parish. So the EP is not in the PARMS. (That is what I have been using to build my PARMS).
If this is correct then the EP is not recorded elsewhere and it has to be entered by the transcriber or proofreader.
Sorry to be contentious but we seem to have differing understandings here.
Geoff
From: Kirk Dawson<EMAIL_ADDRESS>Sent: Thursday, March 26, 2020 1:41 AM
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: geoffj-FUG<EMAIL_ADDRESS>Assign<EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Augmentation of the fields fields transcribed into the initial csv file (#820)
Assigned #820 https://github.com/FreeUKGen/FreeCENMigration/issues/820  to @geoffj-FUG https://github.com/geoffj-FUG .
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/820#event-3164546675 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AKCPIFOKQO4IDRSS3UHBSULRJIQ2HANCNFSM4K5HFGWQ .  https://github.com/notifications/beacon/AKCPIFIRZWI55SCX57ICT4TRJIQ2HA5CNFSM4K5HFGW2YY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOXSPSM4Y.gif

Kirk,
Sorry to sound thick, but I have no instruction at all and don’t know where to start, e.g.  where do I find the SSCENS document or indeed any other documents?
Thanks, Rhoda
From: Kirk Dawson<EMAIL_ADDRESS>Sent: 25 March 2020 15:43
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: Mackenzie, Rhoda<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Augmentation of the fields fields transcribed into the initial csv file (#820)
CAUTION: External email. We are observing an increase in COVID-19 related phishing attempts.
@FreecenBrenhttps://github.com/FreecenBren @geoffj-FUGhttps://github.com/geoffj-FUG @PatReynoldshttps://github.com/PatReynolds @AlOneillhttps://github.com/AlOneill @rhodamackenziehttps://github.com/rhodamackenzie @colin1959https://github.com/colin1959
Since the online upload of a csv file this together with the Params should contain the total set of information required to create a search record now is the time to consider the augmentation of the fields in the csv file.
Rather than include a discussion on each augmentation in this story and end up losing sight of one I would like to suggest a separate story for each.
CEN is in a good position to add new fields in the csv file at it has always used a column name header as the second line in the file. Hence augmentations can be added. This what in REG was termed a flexible csv file. Provided the column header line is included we can be flexible in terms of their order and their number. We can accommodate the situation of absent column header line by assuming the file has the SSCENS layout.
Additional fields that need to be accommodated are:
Ecclesiastical Parish
1901 Census fields
1911 Census fields
Scottish windows field 1851 onwards
deleted flag
detail_flag?
unoccupied_notes
We may need to add others
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/FreeUKGen/FreeCENMigration/issues/820#issuecomment-603911519, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AO5M7OSP4GZGNECUYURWMBDRJIQ6PANCNFSM4K5HFGWQ.
The University of Aberdeen is a charity registered in Scotland, No SC013683.
Tha Oilthigh Obar Dheathain na charthannas clàraichte ann an Alba, Àir. SC013683.

Some docs sent off line

Working on basis that deleted flag and Ecclesiastical Parish are to be included as columns in CSV file
Working on basis that 1901/1911 fields to be added to CSV file

There are 2 additional fields dealing with Birth County and Birth Place See story #840

@geoffj-FUG has put an excellent document together
https://docs.google.com/document/d/1qnWsyBzP5keQscuzpsMaF-CjlDypXJlX/edit

@PatReynolds  and @rhodamackenzie to review.  Kirk has implemented 99% - so focus on what might be missing.

@geoffj-FUG wrote on April 19
I have now almost completed the Fields list for the censuses.
I have done a sweep of England, Wales, Isle of Man, Channel Islands, Scotland and Ireland census forms and identified the information in each of the censuses. I still have a couple of cover sheets to go through but I am 98% there. I expect to receive these within a day or so.
You will now find Version 2 of my New Spreadsheet Sections and fields spreadsheet on Google Docs.
Please go through it and pick out any problems.

I believe that almost all of the fields at the household (dwelling )  and Individual levels are already accommodated; some edits and validations are still required.
There are several new fields not yet supported but these are predominantly associated with Scotland/Ireland Census information level. Are we likely to see actual transcriptions for these in the near future?. In other words do we program them in now or wait till later. The most efficient way is to add now but it will add a few days of coding effort. Guidance from management on this is important
I would plan to hold off code changes until these new fields are finalized. Coding against a moving target is inefficient.

IMHO I think they should be added now, once they done, there is less room for error
Sent from my iPad
On 19 Apr 2020, at 19:11, Kirk Dawson<EMAIL_ADDRESS>wrote:

CAUTION: External email. Ensure this message is from a trusted source before clicking links/attachments.
I believe that almost all of the fields at the household (dwelling )  and Individual levels are already accommodated; some edits and validations are still required.
There are several new fields not yet supported but these are predominantly associated with Scotland/Ireland Census information level. Are we likely to see actual transcriptions for these in the near future?. In other words do we program them in now or wait till later. The most efficient way is to add now but it will add a few days of coding effort. Guidance from management on this is important
I would plan to hold off code changes until these new fields are finalized. Coding against a moving target is inefficient.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/FreeUKGen/FreeCENMigration/issues/820#issuecomment-616198625, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AO5M7ORIOQYSY4CMQQ7UF5TRNM5GJANCNFSM4K5HFGWQ.
The University of Aberdeen is a charity registered in Scotland, No SC013683.
Tha Oilthigh Obar Dheathain na charthannas clàraichte ann an Alba, Àir. SC013683.

My feeling is that we should include them whilst we can. That means that they are there and ready when needed.
Whilst there are embargoes on images by Scotlands People, Ireland images are in the public domain. There are only 2 years 1901 and 1911 as only small fragments of previous years still exist. I see no reason why FreeCEN could not recruit an Irish Coordinator and volunteers when they are ready. Ireland is probably an untapped market and would be viable.
We have the opportunity now. If we don’t do it now it might be ‘too difficult’ when the opportunity arises.
Geoff
From: Kirk Dawson<EMAIL_ADDRESS>Sent: Monday, April 20, 2020 4:12 AM
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: geoffj-FUG<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Augmentation of the fields fields transcribed into the initial csv file (#820)
I believe that almost all of the fields at the household (dwelling )  and Individual levels are already accommodated; some edits and validations are still required.
There are several new fields not yet supported but these are predominantly associated with Scotland/Ireland Census information level. Are we likely to see actual transcriptions for these in the near future?. In other words do we program them in now or wait till later. The most efficient way is to add now but it will add a few days of coding effort. Guidance from management on this is important
I would plan to hold off code changes until these new fields are finalized. Coding against a moving target is inefficient.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/820#issuecomment-616198625 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AKCPIFJTPOWXXIVT56ZF243RNM5GJANCNFSM4K5HFGWQ .  https://github.com/notifications/beacon/AKCPIFK466NWEPTOLJGRHWLRNM5GJA5CNFSM4K5HFGW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOES5HDYI.gif

I am quote sure you know this and I have likely picked you up wrongly @geoffj-FUG but Ireland 1901 and 1911 are all transcribed and available for free http://www.census.nationalarchives.ie/search/ alongside the images

Rhoda
Yes I know that they are available from the Ireland Archives.
I floated the idea a couple of weeks ago that as we were FreeUKGen and Ireland was part of the UK in 1901 and 1911 we should consider their inclusion.
There is nothing to stop the Irish Archives from erecting a pay wall. In that case the transcription would still be on FreeCEN.
Scotland have erected a pay wall through Scotlands People and wont allow further transcription.
Familysearch have the 1881 census and it used to be free. Now they have started to limit access and refer researchers to Findmypast. FreeCEN has started to transcribe 1881 to fill in the gap.
As I did not receive any feedback on Ireland I have continued to include it until I am told to stop. It does not mean we have to transcribe it, it simply means that we have the ability to do so. I am pretty sure that if FreeCEN campaigned for volunteers in Ireland it could become a viable project.
If we don’t do it now it makes it harder in the future.
Geoff
From: Rhoda MacKenzie<EMAIL_ADDRESS>Sent: Friday, April 24, 2020 6:35 PM
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: geoffj-FUG<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Augmentation of the fields fields transcribed into the initial csv file (#820)
I am quote sure you know this and I have likely picked you up wrongly @geoffj-FUG https://github.com/geoffj-FUG  but Ireland 1901 and 1911 are all transcribed and available for free http://www.census.nationalarchives.ie/search/ alongside the images
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/820#issuecomment-618881265 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AKCPIFOQ7OAKJSCFQCBCM3LROFFKTANCNFSM4K5HFGWQ .  https://github.com/notifications/beacon/AKCPIFOOMSN56EY73XOVMKLROFFKTA5CNFSM4K5HFGW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOETRWB4I.gif

I agree that having the capability built in now is good.  In addition to Geoff's reasons, will allow easier searching.

All the fields specified in document
https://docs.google.com/document/d/1CLd_1CsVQcTyjtAFwsmSqQAm8NC02jmV/edit
are now supported in the in the spreadsheet download and the CSV file upload.
Awaiting access to test3 to deploy for testing

Thank you @geoffj-FUG, @rhodamackenzie  and @Captainkirkdawson for the magificent work that is Version 3.
There was one instance of what I think are two different enumerator errors covered in one paragraph (gender and age), I've suggested splitting them.
I have taken a look through it, and it all seems to be as expected, and
I have not proofed, but did notice an inconsistent use of capitalization in one place "transcribers and Proofreaders" - I suggest a find-and-replace exercise to capital ize the roles would fix this.  I have picked up one typo but only because it affected sense (in occupation_category)
Use of Birthplace Chapman Codes:
I don't understand the Cambridge example (Cambridge is in CAM - so whether the enumerator meant Cambridge-the-shire or Cambridge-the-city, the Champan Code is CAM). An example to use here might be Newport, which can be MON or SAL, and therefore should be recorded as UNK (I think, unless the Census is in MON (or surrounding counties) or SAL (or surrounding counties) where it might be presumed that the person from Newport and the enumerator meant the local place???
Verbatim Birth Place
Do we want to specify big-to-small in all cases, not just parished towns, e.g. "Paris, France" is transcribed as France, Paris, and "Newport, Salop" is transcribed as Salop, Newport?
There is a mention of the 'alternative place of birth column' in the 'alternative chapman code' section- but this column doesn't seem to exist.
Notes field: do we now need to specify that there are two notes fields, one to explain the queries, and one for notes for the researcher, e.g. about the potential location of Newport, gender mix up, etc.?
Thank you all again!

Pat
I have replied to the 2 comments on Version 3.
There are a couple of minor changes in my master copy as a result of transcriber feedback.
The Cambridge example was taken from the original FreeCEN instructions. The suffix shire was often not used by enumerators. So Cambridge could refer to Cambridge (the city) or Cambridge (the county). As a transcriber we do not have the information available to us to differentiate between the two. So the original FreeCEN policy was that we enter the larger entity CAM rather that the smaller one Cambridge CAM which could be wrong. Another more obvious instance is a person was born in Dublin. Is the Co Dublin or Dublin town? We do not know. So we enter the Chapman Code DUB and a hyphen. We leave it to the researcher to work out.
Large to small is a good policy. I thought about this a bit. My issue is that I do not know how the FreeCEN gazetteer with grid references is structured. Ideally we should follow that structure to make sure that matching is made more reliable.
The alternative Chapman Code and Place of Birth are just called Chapman Code and Place of Birth, as contrasted with the Verbatim fields. My volunteers who have been using the instructions have raised no issues with these terms, including one new volunteer.
I have added the Newport example to the instructions:
“Another example is Newport with no county enumerated. Newports can be found in MON, NRY, PEM, DEV, SOM, GLS, CON and SAL plus several other Counties, and therefore should be recorded as UNK.”
There is only one notes field on the spreadsheets. Kirk has resolved the conundrum here. I believe the second notes field was generated as part of the reporting system???
Geoff
From: PatReynolds<EMAIL_ADDRESS>Sent: Thursday, May 28, 2020 11:00 PM
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: geoffj-FUG<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Augmentation of the fields fields transcribed into the initial csv file (#820)
Thank you @geoffj-FUG https://github.com/geoffj-FUG , @rhodamackenzie https://github.com/rhodamackenzie  and @Captainkirkdawson https://github.com/Captainkirkdawson  for the magificent work that is Version 3.
There was one instance of what I think are two different enumerator errors covered in one paragraph (gender and age), I've suggested splitting them.
I have taken a look through it, and it all seems to be as expected, and
I have not proofed, but did notice an inconsistent use of capitalization in one place "transcribers and Proofreaders" - I suggest a find-and-replace exercise to capital ize the roles would fix this. I have picked up one typo but only because it affected sense (in occupation_category)
Use of Birthplace Chapman Codes:
I don't understand the Cambridge example (Cambridge is in CAM - so whether the enumerator meant Cambridge-the-shire or Cambridge-the-city, the Champan Code is CAM). An example to use here might be Newport, which can be MON or SAL, and therefore should be recorded as UNK (I think, unless the Census is in MON (or surrounding counties) or SAL (or surrounding counties) where it might be presumed that the person from Newport and the enumerator meant the local place???
Verbatim Birth Place
Do we want to specify big-to-small in all cases, not just parished towns, e.g. "Paris, France" is transcribed as France, Paris, and "Newport, Salop" is transcribed as Salop, Newport?
There is a mention of the 'alternative place of birth column' in the 'alternative chapman code' section- but this column doesn't seem to exist.
Notes field: do we now need to specify that there are two notes fields, one to explain the queries, and one for notes for the researcher, e.g. about the potential location of Newport, gender mix up, etc.?
Thank you all again!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/820#issuecomment-635334293 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AKCPIFJC6UUOAY3CXMSIHXLRTZN3PANCNFSM4K5HFGWQ .  https://github.com/notifications/beacon/AKCPIFISOZT23VLZAHMXC4DRTZN3PA5CNFSM4K5HFGW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEXPG5FI.gif

The FreeCEN gazatteer long/lat (or is it OS references?) is not fit for purpose (that's why we had to take away the 'nearby places' feature.

Thanks Pat
I was not aware of that.
I support the idea of larger to smaller. It helps matching later on if things are in a standard format.
Geoff
From: PatReynolds<EMAIL_ADDRESS>Sent: Wednesday, June 3, 2020 8:43 PM
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: geoffj-FUG<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Augmentation of the fields fields transcribed into the initial csv file (#820)
The FreeCEN gazatteer long/lat (or is it OS references?) is not fit for purpose (that's why we had to take away the 'nearby places' feature.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/820#issuecomment-638113812 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AKCPIFLHNEQ5N2J6MVY3MS3RUYSKXANCNFSM4K5HFGWQ .  https://github.com/notifications/beacon/AKCPIFMDJSC7DWQFIILAEZDRUYSKXA5CNFSM4K5HFGW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEYENQFA.gif

Nearby was not removed because of the FreeCEN gazetter long/lat. BUT because the errors in the rake task that is run as part of the update were not being monitored or corrected! Hence nul location information was left in an increasing number of place and civil districts

Sorry I can’t make tonight as daughters birthday party
Rhoda
Sent from my iPhone
On 6 Jul 2020, at 17:53, PatReynolds<EMAIL_ADDRESS>wrote:

CAUTION: External email. Ensure this message is from a trusted source before clicking links/attachments.
Closed #820https://github.com/FreeUKGen/FreeCENMigration/issues/820.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/FreeUKGen/FreeCENMigration/issues/820#event-3516833024, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AO5M7OQZWZBMT5NREIUINWDR2H6QVANCNFSM4K5HFGWQ.
The University of Aberdeen is a charity registered in Scotland, No SC013683.
Tha Oilthigh Obar Dheathain na charthannas clàraichte ann an Alba, Àir. SC013683.


TPI-1 Invalid sequence_number Check for Mempool in aptos-core

TPI-1 Invalid sequence_number Check for Mempool in aptos-core
Auditor: Movebit
Code: Aptos Core
Severity: Informational
Discovery Methods: Manual Review
Status: Pending
Code Location:
protocol-units/execution/opt-executor/src/executor/transaction_pipe.rs#61-67
Descriptions:
Transactions are added to core_mempool after they have been pre-executed:

The join is checked against the sequence_number of the joining transaction:

Since the db_sequence_number passed in is transaction.sequence_number, this check does
not work, but it is not harmful because of the pre-execution check that was performed
earlier.
Suggestion:
Change the incoming parameter to the real db_sequence_number.

#494


correction in documentation

The dosage file format section reads as, "Columns are snpid rsid position allele1 allele2 MAF id1 ..... idn.", where "snpid" is actually the chromosome number (column 1).

@ScottPDickinson could you resolve this please?

Done.  Thanks!

Thank you @veera-dr and @ScottPDickinson


Provide better documentation

Documentation lacks details. Especially event handling is not clear.

will work on this over the holidays.

Splendid.

please checkout http://mom.js.org/documentations/eventBus.html and give feedback

Cool. Thanks


[STREAMPIPES-433] Set default imagePullPolicy to 'IfNotPresent'

Purpose
This change avoids unnecessary re-pulling of Docker images and thus avoids hitting Docker Hub's pull rate limits
Approach
Simple config parameter change

Hi Dominick,
thanks a lot for the PR.
Can you please change the branch of the PR to dev, since the master is the already released version.

My bad, sorry for the mess

Perfect, thank you. I'll directly merge it.


feat: [Dev branch PR] Runtime + Deployment: Azure function support (preview version)

fixes https://github.com/microsoft/BotFramework-Composer/issues/2924
Preview version of Azure functions deploy support.

Runtime: Tweak and improve AzureFunctions project
Plugin: Minimal changes for function support
Provision script: New version for functions. 99% can be reused, coming in next iteration to avoid unnecesary duplication
ARM template: New arm template for functions equivalent to our web app arm template

Some limitations to the scope and considerations:

Functions can be skills, but ~1 day of work is required for azure functions to be able to be skill hosts.
This should be a preview feature, to gather data about how it is used. If we support multiple runtimes, we might want to be in a position where we don't have to support the deployment mechanisms of the underlying runtimes. More design should come into play around how the user experience is for deployment. How about for more mature projects that have source control?

Experience:

The devops failure seems to be luis related:
0 passing (17s)
  1 failing

  1) Luis Deploy
       can deploy luis success:
     TestingLibraryElementError: Timed out retrying: Unable to find an element with the text: Start Bot. This could be because the text is broken up by multiple elements. In this case, you can provide a function for your text matcher to make your matcher more flexible.
      at Object.getElementError (http://localhost:3000/__cypress/tests?p=cypress/support/index.ts-557:749:19)
      at http://localhost:3000/__cypress/tests?p=cypress/support/index.ts-557:2743:38
      at http://localhost:3000/__cypress/tests?p=cypress/support/index.ts-557:2726:17
      at baseCommandImpl (http://localhost:3000/__cypress/tests?p=cypress/support/index.ts-557:558:16)
      at commandImpl (http://localhost:3000/__cypress/tests?p=cypress/support/index.ts-557:561:40)
      at getValue (http://localhost:3000/__cypress/tests?p=cypress/support/index.ts-557:585:23)
      at resolveValue (http://localhost:3000/__cypress/tests?p=cypress/support/index.ts-557:625:35)


Is ORAS implementation something to consider?

I just found out about this and docker hub has support for it.
Wondering if it's something to consider as part of contajners .
https://github.com/oras-project/

Interesting! Yeah I had known a bit about ORAS sometime back, even before contajners existed in its current form. Do you have an example usecase in mind that you would like to address? I'm asking as that could help in deciding what could be implemented first and more things could be added iteratively.

Not right now, things are quite new.
We manage an artifact repository that we use to deliver releases to environments.
Most of the time we deliver docker images but not all of them are suited to be packaged like a docker image.
If I have time to explore whether it's good enough to package those files as artifacts via docker registry, that might help to keep tooling and process more unified.

My intial thoughts and questions would be:

Would the push pull of the non image artifacts happen in some scripted/code env or via some CLI invocation? I'm asking this as most of the usage patterns of ORAS tooling I'm seeing is via the CLI and contajners isn't meant for direct CLI use, needs embedding.
Are there some exmples of artifacts that could be uploaded to a registry and how would the work flow look like?
Does building of the artifact be covered here too, like we handle container image builds here? ORAS spec dosen't seem to mention of building and if we wanna build it too here, contajners has a pretty hard dependency on container images and Dockerfiles etc.
My gut feeling here is that a separate CLI tool could be a better fit here than contajners which is a library. That tool could use contajners if needed.
Open to any more thoughts and suggestions!

If CLIs are the primary usecase to upload non container image artifacts, we might as well use their official CLI tool and shell out to it via bb or something.

Thanks.
I think it will take some time until things settle down.
I'll keep an eye out - we have some use-cases where storing data in a content addressable repo seems like a good fit.
We will see how things evolve.

If you're okay with it, I can close this and feel free to reopen this with some use cases? Happy to consider them!

Sure, thanks


Make the getScriptLocation function work with mobile & light builds

The current _getScriptLocation function is very strict and will only correctly find the script location when the OpenLayers script is named "OpenLayers.js". This seems highly limiting, especially given that with https://github.com/openlayers/openlayers/commit/c8f0dfd728c6277b52060071338711927b1182e8 :
OpenLayers.debug.js
OpenLayers.light.js
OpenLayers.mobile.js
OpenLayers.light.debug.js  and
OpenLayers.mobile.debug.js
will be built and packaged during the release process and the post commit dev directory update builds.
When _getScriptLocation returns the correct script location, then the default CSS required for OpenLayers UI controls is also included, as is seen in the examples. Without it the CSS file must be specifically included.

Some people would even want a script name that does not include "OpenLayers". Our answer to that has always been "add a  tag referencing OpenLayers' default theme (style.css) to your page, and set OpenLayers.ImgPath. This is mentioned in the Deploying section of the narrative doc: http://docs.openlayers.org/library/deploying.html.

How about this in OpenLayers.js:
_getScriptLocation: function() {
    var scripts = document.getElementsByTagName("script");
    var parts = scripts[scripts.length-1].src.split("/");
    parts.splice(-2);
    return parts.join("/") + "/";
},

And this in SingleFile.js:
_getScriptLocation: function() {
    var scripts = document.getElementsByTagName("script");
    var parts = scripts[scripts.length-1].src.split("/");
    parts.pop();
    return parts.join("/") + "/";
},

Then we can dispense with the scriptName and singleFile stuff.

@tschaub your _getScriptLocation functions rely on the OpenLayers script tag being the last?

@elemoine I understand how not supporting any random names worked and made sense in the past, but now there will be official builds included in the released version that require special treatment versus the full default OpenLayers. When you are building your own and possibly combining it with other libraries or just giving it a non-obvious name, then it is reasonable to expect people to add that additional information. On the other hand, when you are giving people 3 (or technically 6) pre-built options in an effort to lower the bar for beginners and non-geo web developers, then I think it does not make sense to insist on keeping additional, easily mitigated barriers in place.

@elemoine the script executing is the last script in the doc.  Try it.

"Without it the CSS file must be specifically included."
Where's the hardship in that? Nobody expects, for example, jQuery UI to try and work out where the css file is based on script location. It's made clear in the docs that you include a html link.
A bigger problem is the ImgPath still used by some controls and a few other classes (most of which aren't in the light or mobile builds).

@mpriour ...and case sensitive of "OpenLayers.js" is other problems for beginners users. They wonder, why not work?

@probins What I'm trying to point out is that if you are attempting to lower barriers for experimentation and adoption for fresh users, why does everything just work with the full build but not other official included builds. We should either make this work for all official, provided built library versions or we should just remove the automatic CSS & ImgPath logic and make it a requirement to properly configure it for all the official versions.


Enchanting an item with system.attunement required makes you unable to attune to the item

Enchanting an item with system.attunement | override | required makes you unable to manually attune to the item, the actor sheet gets the attunement shortcut button but it's defunct and the attunement checkbox can be saved but does not save.

Duplicate of https://github.com/foundryvtt/dnd5e/issues/3616


Upgrading

In this PR have the following improvements:

[x] Create docs/
[x] Create Makefile
[x] Create adapters/
[x] Create google adapter
[x] Create components/engine.py
[x] Create registry.py
[x] Create trianing_data/
[x] Create typing/
[x] Create utils/io.py
[x] Update README.md information
[x] Update setup.py config
[x] Update examples/
[x] Update of documentation config
[x] Update component
[x] Update dialogue module
[x] Update nlu.py
[x] Remove brain, assistant, dialogue and googleapp

@ralphg6 This is a big PR, should this be merged now?

I think this PR only must be merged when we finish milestone 1.0.0


Getting pm-creds command not found

When running pm-creds command in MacOs I am getting command not found even after adding pm-creds to the PATH variable. Any ideas?

If you have go installed (including the go bin folder in your PATH) you can cd pm-creds && go install.
Did you start a new shell after updating the PATH var also? Or perhaps forgot to run go build so there is no binary built? But easiest is maybe to try the above solution first.

yes it is generating the binary because I can invoke it using the full PATH to the pm-creds and works like that but not doing pm-creds only

Then it has not been added properly to your PATH environment, since it's the PATH variable that controls this behaviour. Or if you're in Windows this procedure might be different from what is explained in the readme.
And you will need to add it to your .zshrc, .bashrc file to have it added each time your start a new shell.
Such as:
export PATH="/your/path/to/pm-creds-bin-folder:$PATH"

Yes I actually see the path doing a echo $PATH but still not working. The path is $HOME/pm-creds/pm-creds/pm-creds

Is that the path to your directly where the pm-creds bin folder is located? It looks like it's an absolute path to the binary file. It should only to the folder, and not the binary. So perhaps try changing it to $HOME/pm-creds/pm-creds:$PATH


Correct mailing list URL. Fixes https://github.com/chaoss/wg-evolutio…

…n/issues/415.
Signed-off-by: lucasgonze<EMAIL_ADDRESS>

Thanks @lucasgonze - looks good


[CELEBORN-1046] Add an expiration time configuration for app directory to clean up

What changes were proposed in this pull request?
Add a configuration "celeborn.worker.storage.expireDirs.timeout" with a default value of 6h in rsswork. This configuration is used to set the expiration time for app local directories.
https://issues.apache.org/jira/browse/CELEBORN-1046
Why are the changes needed?
When Celeborn periodically deletes the directories of apps, it determines whether the app needs to be deleted based on the shuffleKeySet in memory. However, this method may not accurately indicate the completion of the app and could potentially lead to the unintentional deletion of shuffle data.
Does this PR introduce any user-facing change?
How was this patch tested?

Merged to main/0.3


Adding option to make warnings errors and fail running to all tools.

@dwightguth please review.
I don't know how to change the configuration of jenkins to provide the -w2e tag.

Jenkins retest this please

I think I broke Jenkins. Can someone have a look here?
It didn't get my latest commit, and it started giving OutOfMemory exceptions.

Jenkins retest this please

It seems there is something wrong with your code to suppress the flag on certain tests, because it doesn't seem to be working...

I noticed. The problem is that I can't reproduce it on my computer, it only fails on jenkins.

Try testing it on linux. There is at least a tiny amount of platform-specific code in ktest, so it could be because you are running on windows somehow...

lgtm


Inference_inout_memory_opt in inference mode

Problem: Enable inference_inout_memory_opt in inference could make a problem
Cause: If the networks are not flatten which means have a cycle, some of output of layer should be preserved.
For example, in resnet there is a residual output and if the opt is enabled this data could be corrupted.
Solution(?): Check if the layer have a multiple input or output.

:octocat: cibot: Thank you for posting issue #1094. The person in charge will reply soon.

elaborating the yesterday's debugging,
for the resnet, in/out order is not even/odd because the input memory has to be kept to feed the addition layer.
we will need a test for the addition layer inside a model as well.

Memory optimizations have not been tested well with mutli-input/output layers as we didn't have any models using them properly.
This will have to be done soon.


README: Link rather than coping install script shim (address #146)

Here is the PR, in case it turns out to be not such a bad idea.

Closed because it does not make sense as explained in https://github.com/anishathalye/dotbot/issues/146


chore: updating devcontainer to use new recommended settings from microsoft

This PR helps resolves circular imports when specifying type hints for methods. The TYPE_CHECKING flag used is set to True by mypy  while performing type check.
Also fixed the devcontainer based on Emil's cookie cutter template to fix the python interpreter selection issues.

Thanks @efredriksson-modelon  for the review! I have now fixed the pyright warnings. Could you please take a quick look? Thanks! :)


Reset state of results

Ideally, I would need to additions:
[] A way to clear the results array after the recording process is finished
[] Away to trigger 'onResultsReady'

For clearing the results, I can add setResults to the hook return values then you could manually reset the results in a useEffect when isRecording === false
// in your component
useEffect(() => {
  if(!isRecording) {
    setResults([])
  }
}, [isRecording])

As for your second addition, I'm not sure what is meant by triggering onResultsReady, would this be when a new value is added to the results state array? Or the native browser event of a new speech result?
There is no programmatic way to trigger the native result event, this is only fired when the WebSpeechAPI returns a result from captured speech.
If you need to do something when the results state array is updated, you can use a useEffect in your code
useEffect(() => {
  if(results.length > 0) {
    const latestResult = results[results.length - 1]
    const { transcript } = latestResult
  }
}, [results])

If you need access to the native browser result, I can add a new callback prop of onResult that would be invoked on the WebSpeechAPI recognition.onresult event, and when the cross-browser google cloud speech result promise is resolved.
Important to note: The result event would differ depending on if the callback was invoked with WebSpeechAPI or using google cloud API, and you would have to handle parsing the transcript out of the different result events.
I would recommend to keep using the abstracted results state array to reduce complexity unless you really need the native result event

Just put out an update v0.8.0 with the setResults now returned in the hook so you can manually reset


[VS Code] Request was cancelled

Error:
Request was cancelled
Steps to Reproduce:

Version Data

extensionName
VSLS

extensionVersion
0.3.623

protocolVersion
2.2

applicationName
VSCode

applicationVersion
1.26.1

platformName
Linux

platformVersion
4.10.0-42-generic

Thanks for reporting the issue. To help us investigate it, can you please get Live Share logs from VS Code and attach them to the issue? To get the logs, run 'Live Share: Export Logs' command from VS Code command palette. The command will ask you for the zip file location, and will save the logs into that file. You can then attach the file to the issue.

We’re tracking this issue via #469, so I’m going to close this as a duplicate. Thanks!


feat(*): ✨ support node v20, drop support for v14

Pull Request Checklist
🚨 Please review the guidelines for contributing to this repository.

[ ] Appropriate label or labels have been added
[ ] ✅ Tests for the changes have been added (for type:fix / type:feat)
[ ] 📝 Docs have been reviewed, added or updated if needed (for type:fix / type:feat)
[ ] 🏁 Issue number has been linked if this PR closes one

Description

feat(*): ✨ support node v20, drop support for v14

Breaking Change?
BREAKING CHANGE: 💥 node v14 is no longer supported

[x] 💥 Yes
[ ] No

❤ Thank you for contributing!

Codecov Report
Patch and project coverage have no change.

Comparison is base (edf062b) 97.46% compared to head (59de858) 97.46%.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
Additional details and impacted files
@@           Coverage Diff           @@
##           master     #371   +/-   ##
=======================================
  Coverage   97.46%   97.46%
=======================================
  Files          31       31
  Lines         394      394
  Branches       67       67
=======================================
  Hits          384      384
  Misses          8        8
  Partials        2        2

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


attempt to add tweak support

not working.

added support for key tweaking here: https://github.com/jesseposner/FROST-BIP340/commit/700fbd363bd4cca4e96a468a40dc9ae5b17bd7f3


Add dart to language-detector

Hey @szczerski, why did you close it? Does it work"

Hi, I'm just learning how github works. Git-history looks and works great, however, I noticed that the dart language is missing, and now more and more will be written using flutter. That's why I added the same code as for other languages. However, something went wrong.
Sorry for the mess.

@szczerski  I can't reopen it because I think you deleted the fork. Wanna try again in a new PR? I can help you if you need.

My commit would have passed before the test-prettier was added.
The problem appears after the commit 'Add prettier' 941a28ad39703.

That should be easy to fix, you need to run yarn format and commit the changes. You can also install prettier in your code editor, it's an amazing tool

Thanks for the hints @pomber . I changed the prettier path to
\"src/**/*.{js,jsx,md,json,html,css,yml}\" or without the quotes *.{js,jsx,md,json,html,css,yml} and pass the tests. I am now on Windows, maybe tomorrow I'll check if Linux will pass tests without changing the path.

This time, it passes (PR #82) and linux has no problems with the path.


Cannot Place UI elements over viewport

I can't figure out how to draw some Ui elements over the map.
I was using google maps and cordova previously and I could "draw" over a map since it was drawn within a div container.
In this case I don't know which element, if any I can apply some css to in order to draw over the map. Is it even possible to do so?

Using a GridLayout you can place elements on top of the MapBox now if you declare it in your markup.
If you need an idea how to 'float' a view component on top of another check the FAB plugin README here using a GridLayout https://github.com/bradmartin/nativescript-floatingactionbutton#xml


Trouble finding error...[slovenian] sentences

Hi!
I do my best to translate sentences into Slovenian language and now I'm in a phase to test with python script:
python3 -m script.intentfest parse --language sl --sentence 'vključi luč na terasi'
I get this:
{   "text": "vključi luči na terasi",   "match": false }
Since I have trouble with other sentences too, I assume there is some other issue...maybe with insufficient translations of other parts or cells...
Can someone point me towards the right direction? Maybe @spuljko since I used most of Croatian translations (which btw are working in test)? Or any other tip how to checklist the areas maybe?
Thank you in advance!

I don't know any Slovenian and you haven't provided enough info (regarding an actual commit) for me to actually give you a definitive answer, but with the help of Google Translate and considering that you're referring to one of your many pull requests as of late, I will assume that you don't have an area named terasi in your _fixtures.yaml, which causes the intent not to be recognized.
May I also recommend the #devs_voice channel in the Discord server for such questions.

You are using singular in test and sentence expects plural (luč|luči). You can change sentence like this

"<vključi> (luč[i]|svetil(a|o)|razsvetljavo|svetilk(u|e)) (v|na) [sobi] "

Pay attention to brackets, what is in square brackets is not requred in sentence, while expression in round brackets is required. More info here: https://developers.home-assistant.io/docs/voice/intent-recognition/template-sentence-syntax/
Also, please don't create pull request for every push and divide branches in more logical way - for example one for lights, one for covers etc. Name PR so that everyone can understand what is changed, like "[si] addded lights sentences". Don't put things from one PR to another, always start branch from main.
More info here under Creating perfect PR: https://developers.home-assistant.io/docs/review-process/
Hope that helps, feel free to ask if you need anything else.

you don't have an area named terasi in your _fixtures.yaml, which causes the intent not to be recognized.

Good point, I didn't check fixtures.

Thank you @tetele and @spuljko! Very helpful! I was checking sentences and not other files. I did work on files and sentences, I found _fixtures.yaml only in tests folder.
If I run:
python3 -m script.intentfest parse --language sl --sentence 'Vklopi luč v kuhinji'
I still get:
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/github2/intents/script/intentfest/__main__.py", line 29, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/github2/intents/script/intentfest/__main__.py", line 24, in main
    return module.run()
           ^^^^^^^^^^^^
  File "/home/github2/intents/script/intentfest/parse.py", line 89, in run
    render_response(response_template, result, matched, unmatched)
  File "/home/github2/intents/shared/__init__.py", line 159, in render_response
    assert response_template, "Empty response template"
AssertionError: Empty response template

I corrected _fixtures.yaml like this:
language: sl
areas:
  - name: "spalnic(a|i|o)"
    id: bedroom
  - name: "garaž(a|i|o)"
    id: garage
  - name: "pisarn(a|i|o)|kabinet(a|u)"
    id: office
  - name: "balkon(u)"
    id: balcony
  - name: "teras(a|i|o)"
    id: terrace
  - name: "omar(a|i|o)"
    id: closet
  - name: "vhod(u|om)|predverj(e|u)"
    id: entrance
  - name: "hodnik(u|ih)"
    id: hallway
  - name: "shramb(a|i|e|ah)"
    id: pantry
  - name: "garderob(a||i|u)"
    id: wardrobe
  - name: "stranišč(e|u|ih)|(v|w)c[ju]"
    id: wc
  - name: "stopnišč(e|u|ih)|stipnic(e|ah|i)"
    id: staircase
  - name: "kuhinj(a|i|e|jah|ski|ska|ske|skih)"
    id: kitchen
  - name: "dnevna soba|dnevni sobi|dnevne sobe"
    id: living_room
  - name: "kopalnic(a|i|o|ah|ama)"
    id: bathroom
  - name: "utility"
    id: utility
entities:
  - name: "nočn(a|e|i) (svetil(ka|ke|a|o)|svjetl(o|a)|rasvjet(a|i|u)|luč(ka|i|ke|ki))"
    id: light.bedroom_lamp
    area: bedroom
    state: "off"

  - name: "luč(i|ke) na terasi"
    id: light.terrace
    area: terrace
    state: "off"

  - name: "luč(i|ke) na balkonu"
    id: light.balcony
    area: balcony
    state: "off"

  - name: "kuhinjs(ko|ka|ki) stikal(o|la|li)|tipk(a|lo|li)"
    id: switch.kitchen
    area: kitchen
    state: "on"

  - name: "stropn(i|a|e|nemu) ventilator(ja|je|ju)"
    id: fan.ceiling
    area: living_room
    state: "on"

  - name: "lesten(ec|ca|ce)"
    id: light.chandalier
    area: living_room
    state: "on"

  - name: "božične luč(ke|i|ki|ko)|"
    id: light.chrismas_ligths
    area: living_room
    state: "on"

  - name: "taln(o|a|e|i) svetilk(a|e|i|o)"
    id: light.floor_lamp
    area: living_room
    state: "on"

  - name: "[stropni] ventilator [na stropu]"
    id: fan.ceiling
    area: living_room
    state: "on"

  - name: "[kuhinjsk(a|o)] (napa|napo)"
    id: fan.kitchen_hood
    area: kitchen
    state: "on"

  - name: "lev(a|o|i|e) zaves(a|i|o|e)"
    id: cover.curtain_left
    area: living_room
    state: "open"
    attributes:
      device_class: curtain

  - name: "dvoriščn(a|i|e) vrat(a|i|e)"
    id: cover.main_gate
    area: "entrance"
    state: "closed"
    attributes:
      device_class: gate

  - name: "garaž(a|na|ne|i|o)"
    id: cover.garage_doors
    area: "garage"
    state: "closed"
    attributes:
      device_class: garage

  - name: "balkonsk(a|e|i) rolet(e|a|i)"
    id: cover.balcony_blinds
    area: "living_room"
    state: "closed"
    attributes:
      device_class: blinds

  - name: "kopalnišk(o|a|i) okn(a|o|i)"
    id: cover.bathroom_window
    area: "bathroom"
    state: "open"
    attributes:
      device_class: window

  - name: "strešn(a|o|i) okn(o|i|a)"
    id: cover.ceiling_window
    area: "kitchen"
    state: "open"
    attributes:
      device_class: window

  #door
  - name: "vhodn vrata"
    id: cover.entrance_door
    area: "entrance"
    state: "closed"
    attributes:
      device_class: door

  - name: "kuhinjska vrata"
    id: cover.kitchen_door
    area: "entrance"
    state: "open"
    attributes:
      device_class: door

  - name: "drsna vrata"
    id: cover.living_room_door
    area: "living_room"
    state: "open"
    attributes:
      device_class: door

  - name: "razvlaževalec|razvlažilnik (zraka)"
    id: "humidifier.dehumidifier"
    area: "living_room"
    state: "off"

  - name: "radiator|radiatorji"
    id: "climate.trv_living_room"
    area: "living_room"
    state: "21"
    attributes:
      unit_of_measurement: "°C"

  - name: "zunanja temperatura|[temperatura] zunaj"
    id: "sensor.outside_temperature"
    state: "17"
    area: "zunaj"
    attributes:
      unit_of_measurement: "°C"

  #locks
  - name: "vhodna vrata"
    id: "lock.front_door"
    area: "entrance"
    state: "locked"

  - name: "vhodna vrata"
    id: "lock.front_door"
    area: "predsoba"
    state: "locked"

  - name: "Zadnja vrata"
    id: "lock.back_door"
    area: "kitchen"
    state: "unlocked"

  - name: "Dvoriščna vrata"
    id: "lock.yard_door"
    state: "locked"

  - name: "Dovozna vrata"
    id: "lock.yard_gate"
    state: "locked"

  - name: "Igranje"
    id: "scene.play_mode"

  - name: "sesaj|posesaj kuhinjo"
    id: "script.vacuum_clean_kitchen"

In light_HassTurnOn.yaml I have (I went for Croatian way:):
language: sl
intents:
  HassTurnOn:
    data:
      - sentences:
          - Vklopi luči v dnevni sobi
        intent:
          name: HassTurnOn
          slots:
            area:
              - dnevni sobi
            domain: light
        response: "Luči so vklopljene"

      - sentences:
          - "Vklopi luč v kuhinji"
        intent:
          name: HassTurnOn
          slots:
            area:
              - kuhinji
            domain: light
        response: "Luč je vklopljena"

      - sentences:
          - Vklopi luči na hodniku
        intent:
          name: HassTurnOn
          slots:
            domain: light
            area:
              - hodniku
        response: "Luči so vklopljene"

So, I assume I should check responses...and I referenced from Croatian file HassTurnOn and changed it accordingly:
language: sl
responses:
  intents:
    HassTurnOn:
      default: "Vklapljam {{ state.domain }}" #default: Naprava {{ slots.name }} je vklopljena
      default_name: "Vklapljam {{ slots.name }}"
      lights_area: "Luč je vklopljena" #lights_area: Luči v {{ slots.area }} so vklopljene
      fans_area: "Ventilator je vklopljen" #fans_area: Ventilatorji v {{ slots.area }} so vklopljeni
      cover_device_class: "Odpiram {{ slots.device_class }}" #cover_single: "Odpiram {{ slots.name }}" #cover: Senčilo {{ slots.name }} je odprto
      cover_single: "Odpiram {{ slots.name }}" #cover_area: Senčila v {{ slots.area }} so odprta
      lock: "Zakljenjena"
      scene: "Aktivirana"
      script: "Zagnana"

comments in file is the old config, as I said I try to head the HR way because it works.
One thing I don't understand in light_HassTurnOn.yaml I have this sentence configured:
- sentences:
          - "Vklopi luč v kuhinji"
        intent:
          name: HassTurnOn
          slots:
            area:
              - kuhinji
            domain: light

Why is the area: like - kuhinji and not - kuhinja ? Is this OK?
file _common.yaml :
language: "sl"
responses:
  errors:
    no_intent: "Oprosti, tega nisem razumel"
    no_area: "Območje z imenom {{ area }} ne obstaja"
    no_domain: "V območju {{ area }} ne obstaja {{ domain }}"
    no_device_class: "V območju {{ area }} ne obstaja {{ device_class }}"
    no_entity: "Ni naprave ali entitete z imenom {{ entity }}"
    handle_error: "Med izvajanjem je prišlo do nepričakovane napake"
lists:
  color:
    values:
      - in: "bela"
        out: "white"
      - in: "črna"
        out: "black"
      - in: "rdeča"
        out: "red"
      - in: "oranžna"
        out: "orange"
      - in: "rumena"
        out: "yellow"
      - in: "zelena"
        out: "green"
      - in: "modra"
        out: "blue"
      - in: "vijolična"
        out: "purple"
      - in: "rjava"
        out: "brown"
  brightness:
    range:
      type: "percentage"
      from: 0
      to: 100
  temperature:
    range:
      type: "temperature"
      from: 0
      to: 100
  temperature_unit:
    values:
      - in: "C | stopinj[a|i|e] [Celzija]"
        out: "celsius"
  brightness_level:
    values:
      - in: (max|maximum|highest)
        out: 100
      - in: (minimum|lowest)
        out: 1
  on_off_states:
    values:
      - in: "vklopljen[a|o|i]|vključen[a|o|i]"
        out: "on"
      - in: "izklopljen[a|i]|izključen[a|i|o]"
        out: "off"
  on_off_domains:
    values:
      - in: "luč[i]|svetilo[a]"
        out: light
      - in: "ventilator[ji|ja]|ventilacija[je]|prezračevanje"
        out: fan
      - in: "stikalo[a|i]|gumb[a|i]|tipka[i|e]"
        out: switch
  cover_states:
    values:
      - in: "odprt[a|i|o]|dvignjen[a|i|o]"
        out: "open"
      - in: "zaprt[o|a]|spuščen[a|i|o]"
        out: "closed"
      - in: "odpiram"
        out: "opening"
      - in: "zapiram"
        out: "closing"
  cover_classes:
    values:
      - in: pergol(a|e|o|i)|tend(a|e|o|i)|nadstrešnic(a|o|i|e)
        out: awning
      - in: rolet(a|e|i|o)|senčil(o|a|i)|senčnik[a|i]|rol[o|ji|ja]
        out: blind
      - in: zavesa(a|e|o)|zastor[a|i]
        out: curtain
      - in: vrata
        out: door
      - in: garažna vrata
        out: garage
      - in: vrat(a|i)
        out: gate
      - in: žaluzij(a|e|o|e)
        out: shade
      - in: zapah[i|e]|pokrov
        out: shutter
      - in: okn[a|i]
        out: window
  lock_states:
    values:
      - in: "zaklenjen[a|i|o]"
        out: "locked"
      - in: "odklenjen[a|i|o]"
        out: "unlocked"
expansion_rules:
  name: "{name}"
  area: "[v [prostoru] | na] {area} [sobi]"
  brightness: "{brightness} [procent[a|e|ov] | odstot[e]k[a|e|ov]]"
  temperature: "{temperature}[°| <stopinj>] [{temperature_unit}]"
  stopinj: "[stopinj[a|i|e]]"
  temp: "(stopinj[a|i|e] | temperatur[a|i|e|o])"
  set: "(nastavi | spremeni | prilagodi | segrej | ohladi)"
  close: "(zapri | spusti | zagrni | zatesni | z[a]vij)"
  open: "(odpri | dvigni | odgrni | razkrij | razvij)"
  covers: "([garažna | vhodna | balkonska | zadnja | sprednja | drsna]) vrata | [strešn(o|i|a)] okn(o|i|a) | rolet(o|i|e) | žaluzij(o|i|e) | markiz(a|i|e|o) | tend(a|i|e|o) | [pokrov | pokrivalo] bazena | ograj(a|i|e|o)"
  vklopi: "vključi[te|mo]|vklopi[te|mo]"
  vključi: "vključi[te|mo]|vklopi[te|mo]"
  izklopi: "izključi[te|mo]|ugasni[te|mo]"
  izključi: "izključi[te|mo]|ugasni[te|mo]"
  spremeni: "prilagodi[te|mo]|nastavi[te|mo]|zamenjaj[te|mo]|zmanjšaj[te|mo]|povečaj[te|mo]"
  odpri: "odpri[te|mo]|dvigni[te|mo]"
  zapri: "zapri[te|mo]|spusti[te|mo]"
  katera_je: "katera je|povej [mi]|katerakoli|katero"
  kakšna_je: "kakša je|povej [mi]"
  vse: "[vsi|vsa|vse]"
  lock: "vrata"
skip_words:
  - "prosim"

Can't really find the reason besides that I translate the rest :)
Guys, I'm really glad for your help!!!!

Don't use sentence syntax in fixtures. The fixtures are basically a simulation of what HA would provide to the intent recognition component and since you don't have names like teras(a|i|o) for an area in HA, you shouldn't have them in fixtures either.
These are only used for tests. It's ok to be a little bit more rigid, as it's only a test.

Thanks @tetele, I corrected to kuhinja and for whis I tested it with:
python3 -m script.intentfest parse --language sl --sentence 'vklopi luč v kuhinji'

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/github2/intents/script/intentfest/__main__.py", line 29, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/github2/intents/script/intentfest/__main__.py", line 24, in main
    return module.run()
           ^^^^^^^^^^^^
  File "/home/github2/intents/script/intentfest/parse.py", line 89, in run
    render_response(response_template, result, matched, unmatched)
  File "/home/github2/intents/shared/__init__.py", line 159, in render_response
    assert response_template, "Empty response template"
AssertionError: Empty response template

And if I test it with 'vklopi luč v kuhinja'
I get:
{
  "text": "vklopi luč v kuhinja",
  "match": false
}

Please read the documentation on how to write intents.
This is how you include placeholders in your sentences
This is how you define a response
In your case, in sentences/sl/light_HassTurnOn.yaml, you should probably have something less hard-coded, like:
language: sl
intents:
  HassTurnOn:
    data:
      - sentences:
          - "Vklopi luč v {area}"
        intent:
          name: HassTurnOn
          slots:
          #   area: # <- i've commented these out, but you should remove them, as the {area} is dynamically filled from fixtures
          #     - kuhinji
            domain: light
        response: default # <- notice the key from responses/sl/HassTurnOn.yaml

Read up on the documentation for contributing and check out the structure defined for English in sentences/en, responses/en and tests/en

I just noticed you don't have any light attached in the kitchen in your fixtures, so even with the modifications above, there's nothing to turn on.
A test should sound like this: "Given the stuff defined in tests/sl/_fixtures.yaml with fixed names, if I say any of the the exact sentences X1..Xn, I expect the exact response Y and the slots Z1..Zn"
To test that, the recognizer will go through all of the sentences you defined in sentences/sl, provide them with the predefined slots name and area, try to match any one of them given the constraints defined through requires_context and excludes_context, then see what response key the match offers. After that, it goes to the responses/sl/WhateverMatchedIntent.yaml and gets the response text, substitutes the placeholders with actual matched entity info (e.g. name, area, slots etc.) and returns the response.
If the returned response matches your expected response, the test will pass.

Aha, thank for the logic behind it @tetele ! Tried to go through the large amount of docs since I'm quite a noob, sorry...
I'll get back to you and report :)
Have a nice day!

Define "luč(i|ke) na terasi" as "luč[i|ke] na terasi" so it will match luč, luči and lučke.
Try to name your entities differently, maybe it will help you figuring stuff out - instead of "luč na terasi" name it for example "solarn(e|o) luč[i|ke". That way you will know if you are calling area sentences or entity sentence (vkopi luči na terasi -> area - that will turn all lights, vklopi solarne lučke -> entity - that will only turn on that named entity. you can also write "vklopi solerne lučke na terasi")

OK, thank you @tetele @spuljko ! I'm having some progress. My first sucessful test 👍 :
python3 -m script.intentfest parse --language sl --sentence 'vklopi luč na terasi'
and output:
{
  "text": "vklopi luč na terasi",
  "match": true,
  "intent": "HassTurnOn",
  "slots": {
    "name": "luč na terasi"
  },
  "context": {
    "domain": "light"
  },
  "response_key": "default_name",
  "response": "Vklapljam luč na terasi"

@tetele @spuljko thank you again so much for help and guidance with my first PR and commit :).
For me as unexperienced in github wokflow and in coding the learing curve is very steep :) but I'll try to do my best not to bother you too much.
Thanks for relevant docs!

We all started someplace.
Don't forget to add tests to your ever-growing PR. When I see tests added, I'll approve the CI run on your PR.

From here

Read up on the documentation for contributing and check out the structure defined for English in sentences/en, responses/en and tests/en

Reading here
but can't find out of I should c/p files from sentences dir to tests dir, the link is broken.

@tetele or @spuljko I kindly ask for help...
I read the docs and try to figure out the responses. In [en] responses file is following:
language: en
responses:
  intents:
    HassLightSet:
      brightness: '{{ slots.name }} brightness set to {{ slots.brightness }}'

But in actual responses file from repo is following:
language: en
responses:
  intents:
    HassLightSet:
      brightness: "Brightness set"
      color: "Color set"

So in actual response file there is no template used...but is there any other relevant difference between there two files?
I try to understand what't wrong with my sentences and I try to understand this with "reverese" engineering :).
Second question:
WARN] responses/sl/HassTurnOn.yaml: unused response default Has  default  also be provided? Is there any example?
Big thanks to any suggestion and hint :)!!!

You can use templates in response (check other files), but I would recommend to keep it as simple as you can, or you will find you in grammar rules hell.
Warning is that you have response you are not using in sny sentence, you can delete that response if you don't need it.

Thanks @spuljko . Getting some progress after 2 days of beeing stuck with light_HassLightSet sentence. Started test, sentence and response file from only 1 sentence...and finally test passed!!!
No clue, how the light_HassLightOff did pass it, with very similar structure, but I'm very happy I did it.
I got now from here with small steps :)) Thank you again for support! It means a lot! 🥇

In [en] responses file is following:

That's just an example in the docs, it's not the EN responses file. Anyway, glad you figured it out


SIGSEGV in PhaseIdealLoop::build_loop_late_post_work

Summary

We run an Eclipse-based product obfuscated using ZKM. Running our tests on CI has been causing frequent SIGSEGV's.
Steps to reproduce

The error is "rare" (one in 10 builds usually, on some of our branches it's 3/4, on others 1/20),
See our hs_err_pid, replay_pid and core dump (2.3 GB)
We're determined to help you help us. If there's anything more we can do, please let us know. We're trying to minimize this to a reproducible example, but that will take time, and definitely won't be easy due to the extreme flakiness of the failure.
Expected results
No crash
Actual results
Random SIGSEGV's likely heavily influenced by code layout and timings.
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007ff7086a270b, pid=27153, tid=27169
#
# JRE version: OpenJDK Runtime Environment Temurin-17.0.3+7 (17.0.3+7) (build 17.0.3+7)
# Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.3+7 (17.0.3+7, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0xac870b]  PhaseIdealLoop::build_loop_late_post_work(Node*, bool)+0x13b
#
# Core dump will be written. Default location: /home/jenkins/agent/workspace/line_8382-speed-up-product-build/com.sigasi.hdt.vhdl.test.projects/core.27153
#
# An error report file with more information is saved as:
# /home/jenkins/agent/workspace/line_8382-speed-up-product-build/com.sigasi.hdt.vhdl.test.projects/hs_err_pid27153.log
#
# Compiler replay data is saved as:
# /home/jenkins/agent/workspace/line_8382-speed-up-product-build/com.sigasi.hdt.vhdl.test.projects/replay_pid27153.log
#
# If you would like to submit a bug report, please visit:
#   https://github.com/adoptium/adoptium-support/issues
#

Triaging info
Java version:

OpenJDK Runtime Environment Temurin-17.0.3+7 (17.0.3+7) (build 17.0.3+7)
What is your operating system and platform?

Amazon Linux release 2 (Karoo) on x86-64
How did you install Java?

Binary archive, tar.gz.
Did it work before?

We've been having this issue for months, since it relies on timings and code layouts there are periods in which we have many failures, then weeks without any.
Did you test with other Java versions?
Been having this since < Java 17. We haven't tried other VMs such as Graal or OpenJ9.
We've been faithfully upgrading to the latest Temurin version since Java 11, up to 12, 13, 14, 15 and now 17. (we skipped 16).

We don't use Lucene nor JavaFX, but that is the only link I found to PhaseIdealLoop::build_loop_late_post_work as well

In the meantime, the linked bug has been split in two. The first appears irrelevant to us, we still encounter the issue with JustJ 17.0.4. The second is probably the one we need fixing.

<IP_ADDRESS> should have https://bugs.openjdk.org/browse/JDK-8275610 fixed (the first)

Pinged about this at EclipseCon by @tivervac

Maintainers are commenting on the upstream issue, but it's still open for now.

@uschindler That's quite possible. By now we've been able to (temporarily?) work around this issue by not obfuscating one of our classes. We obfuscate using ZKM. That obfuscator likes to split off code and wrap it in other classes.

So you don't have the source code of:
C2:  25179 18398       4       com.sigasi.hdt.vhdl.effectanalysis.d::visitIdentifierPathElement (97 bytes)
??? Too bad. Maybe a disassembly of bytecode?

Sadly, I had this info at the time of writing, but not anymore at this point.
I'll see whether I can reproduce it again

Am I correct that upstream bug should be resolved in 19.0.2+7 ?
I feel that we faced same problem using 19.0.2+7

No fix version is for Java 20

@karianna How can I check it?
I thought that https://bugs.openjdk.org/browse/JDK-8297510 corresponds to issue above and should be resolved in 19.0.2 build 7

https://bugs.openjdk.org/browse/JDK-8285835

The link shows that the issue was backported

Fair point, should be fixed then.  If you have a new crash log I can post it.

@vans239 Are you able to try the latest LTS or Java 20.0.1 and let us know if this is resolved for you?

Sadly even with the backported bug fix mentioned above, we're still encountering the issue with Temurin 17.0.7+7

Are you able to try the latest LTS or Java 20.0.1 and let us know if this is resolved for you?

We were still observing issues with 19.0.2+7. I am trying currently 20.0.1+9 and not able reproduce so far. Will have more info next week when we rollout it fully to prod

Sadly even with the backported bug fix mentioned above, we're still encountering the issue with Temurin 17.0.7+7

I would try 17.0.8 JIC.

Maybe related: https://github.com/openjdk/jdk/pull/15399 ... "SIGSEGV in PhaseIdealLoop::build_loop_late_post_work" ... but for a different reason. Backporting has not been even discussed on that one yet.

Definitely possible, thanks for the link!

Here are 2 crashes from Apache Pulsar, full hs_err_pid*.log files: https://gist.github.com/lhotari/53b72683ad4f339dfbcfd8b9b97062b9 .
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f927e8d5113, pid=3924, tid=4012
#
# JRE version: OpenJDK Runtime Environment Temurin-<IP_ADDRESS>+1 (<IP_ADDRESS>+1) (build <IP_ADDRESS>+1)
# Java VM: OpenJDK 64-Bit Server VM Temurin-<IP_ADDRESS>+1 (<IP_ADDRESS>+1, mixed mode, sharing, tiered, compressed class ptrs, z gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0xad5113]  PhaseIdealLoop::build_loop_late_post_work(Node*, bool)+0xe3
#

Happens in <IP_ADDRESS> . The Apache Pulsar issue is https://github.com/apache/pulsar/issues/19307 . Any help is appreciated.

Looks like the previously posted GH issue links to https://bugs.openjdk.org/browse/JDK-8314024 which will be backported to 17.0.10 .


Update to Breez SDK v0.3.3

https://github.com/breez/breez-sdk/releases/tag/0.2.14

https://github.com/breez/breez-sdk/releases/tag/0.3.3
needed to resolve closed channels issue https://github.com/lnflash/flash-mobile/issues/97


Change the order of downloading config files and sourcing env files

Currently, Genie sources env files [1], and then downloads config files [2]. But it would make more sense if we changed the order to [2] and [1]. Here is why:
I am launching a Spark job via Genie. I wanted to create a single Spark application and link two Spark commands with different configs (for eg, prod and test) to the application. I also wanted to download config files and set SPARK_CONF_DIR in the env files. But since sourcing env files happens prior to downloading config files, I cannot do this now. As a workaround, I ended up creating two Spark applications.
I think setting XX_CONF_DIR on the fly is a common practice in Hadoop and Spark. So it makes sense to download config files before sourcing env files.

NetflixOSS » genie » genie-pull-requests #3 SUCCESS
This pull request looks good


.ma ccTLD support

No support for .ma ccTLD.
>>> print(w)
{
  "domain_name": null,
  "registrar": null,
  "whois_server": null,
  "referral_url": null,
  "updated_date": null,
  "creation_date": null,
  "expiration_date": null,
  "name_servers": null,
  "status": null,
  "emails": null,
  "dnssec": null,
  "name": null,
  "org": null,
  "address": null,
  "city": null,
  "state": null,
  "registrant_postal_code": null,
  "country": null
}
>>> w.text
'% This is the RIPE Database query service.\n% The objects are in RPSL format.\n%\n% The RIPE Database is subject to Terms and Conditions.\n% See http://www.ripe.net/db/support/db-terms-conditions.pdf\n\n% Note: this output has been filtered.\n%       To receive output for a database update, use the "-B" flag.\n\n%ERROR:101: no entries found\n%\n% No entries found in source RIPE.\n\n% This query was served by the RIPE Database Query Service version 1.105 (DEXTER)\n\n\n'

PR would be welcome!

Closing as no reply


E2e test with cypress

This is a version that works for login and upload weedcoco, but I reckon it's not the most elegant way doing it. I tried a bit from https://github.com/testing-library/cypress-testing-library/blob/97939da7d4707a71049884c0324c0eda56e26fc2/cypress/integration/find.spec.js but didn't make it to locate element easily like finding the sign in button even.

There also seems to be a dep version conflict between node and cypress.

This is now complaining that 001_image.png is missing. Did you forget to commit it? (perhaps use a low-quality jpeg to save space??)

I should clarify: it turns out that the celery and django jobs failed, since .env didn't exist. This is now rectified.
#457 improves handling such errors so that they lead to CI failure.
afc40d98 halved the runtime of the frontend.yml check by caching layers in the docker build.

It passes 🥂🥳🎉

Okay to take this out of Draft mode?


twister: make skip reason clearer

The aim of this PR is to make the skip reasons less obscure. Without it, there is a wide class of configurations getting "Not runnable on device" status, which is quite obscure. There are several different reasons for this status: harness not supported, fixture not available, missing installation of a simulator, etc. Statuses are clearer with this PR
Before:
scripts/twister  -T samples/boards/nrf/clock_skew -T samples/basic/blinky -T tests/drivers/gpio/gpio_basic_api -v -v --inline-logs -p nrf52840dk_nrf52840 --device-testing --device-serial /dev/ttyACM0 :
DEBUG   - nrf52840dk_nrf52840       samples/boards/nrf/clock_skew/sample.boards.nrf.clock_skew SKIPPED: Not runnable on device
DEBUG   - nrf52840dk_nrf52840       samples/basic/blinky/sample.basic.blinky           SKIPPED: Not runnable on device
DEBUG   - nrf52840dk_nrf52840       tests/drivers/gpio/gpio_basic_api/drivers.gpio.2pin SKIPPED: Not runnable on device
DEBUG   - nrf52840dk_nrf52840       tests/drivers/gpio/gpio_basic_api/drivers.gpio.nrf_sense_edge SKIPPED: Not runnable on
and
scripts/twister -T samples/hello_world -p nsim_hs -p hifive1 -v -v --filter runnable:
DEBUG   - nsim_hs                   samples/hello_world/sample.basic.helloworld        SKIPPED: Not runnable on device
DEBUG   - hifive1                     samples/hello_world/sample.basic.helloworld        SKIPPED: Not runnable on device
With the PR the above commands produce:
DEBUG   - nrf52840dk_nrf52840       samples/boards/nrf/clock_skew/sample.boards.nrf.clock_skew SKIPPED: Not runnable on device: test is marked as build-only.
DEBUG   - nrf52840dk_nrf52840       samples/basic/blinky/sample.basic.blinky           SKIPPED: Not runnable on device: Harness led is not supported.
DEBUG   - nrf52840dk_nrf52840       tests/drivers/gpio/gpio_basic_api/drivers.gpio.2pin SKIPPED: Not runnable on device: Fixture gpio_loopback is not available.
DEBUG   - nrf52840dk_nrf52840       tests/drivers/gpio/gpio_basic_api/drivers.gpio.nrf_sense_edge SKIPPED: Not runnable on device: Fixture gpio_loopback is not available.
and
DEBUG   - nsim_hs                   samples/hello_world/sample.basic.helloworld        SKIPPED: Not runnable on device: nsimdrv not found.
DEBUG   - hifive1                     samples/hello_world/sample.basic.helloworld        SKIPPED: Not runnable on device: renode not found.
Each commit from this PR is explaining the changes introduced.

are you going to push an update to pass CI?

Yes. I've done quite some local testing with various scenarios and I hope I managed to get the logic right. I didn't check yet if the tests/samples failing in twister-build stage are PR related. There are for sure tests of twister itself (pytest) failing and they will require updating, since they use functions I modified and they have to be adapted to this modified logic.
Unluckily, I won't be able to properly analyze at fix it soon, since I will be on holidays. I will come back to it in 2 weeks. Do you prefer closing this PR until I have time to get back to it?

Do you prefer closing this PR until I have time to get back to it?

no. I was just wondering if I should start revieweing or if there is an update coming to fix CI

I rebased the PR and added a WIP commit where I started to update twister tests. At least we will see if the failures in twister-build were/are PR related.

@gopiotr Thanks for the comments, I applied your suggestions and force-pushed the PR.

@nashif Can you give it another look? I addressed all the issues found by you and Piotr and I explained the motivation behind the changes.

@PerMac  I am fine with making the skip reason clearer, but changing the workflow and applying filter on every call is not related and can have very serious consequences. So it need to come as a seperate PR after putting more thought. I do not think we should be changing the workflow and how we filter tests as a side effect of what seems like a harmless enhancement of making skip reason clearer.