Migration table not created in designated schema

The migration table is created inside the public schema regardless of the schema defined in the database config file when migrating to redshift. I want to be able to house multiple schemas on the same DB in redshift, and have separate migrations per schema. Is this possible?
Below is the verbose output from db-migrate when performing a migration on a redshift database:
[INFO] connected
Inside create migrations table
[SQL] select version() as version
[SQL] SHOW search_path
[SQL] SET search_path TO "dev","$user"," public"
[SQL] SELECT table_name FROM information_schema.tables WHERE table_name = 'migrations'
[INFO] migration table created
[INFO] loading migrations from dir /Users/Josh/Projects/Compass.Web/database/migrations/warehouse
[INFO] loading migrations from database
[SQL] SELECT * FROM "migrations" ORDER BY run_on DESC, name DESC
[INFO] No migrations to run
[INFO] Done
It looks like it captured the schema, and sets the search path to it, but then doesn't pick up the correct migration table.

---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/25497254-migration-table-not-created-in-designated-schema?utm_campaign=plugin&utm_content=tracker%2F73887&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F73887&utm_medium=issues&utm_source=github).

Please provide all necessary informations to reproduce this.
I'm currently missing:
db-migrate version used,
node version used

Also I could be wrong, but I think this problem was also solved in the new version which was released today (v0.9.19), which includes the following bugfix which I merged today:
#287
Please always include the version information and all necessary informations to complete reproduce the issue. So we can help you way faster, without having to guess and collect data :)

@wzrdtales You are correct, this is basic information that I did not mean to leave out.
Node version - 0.10.38
db-migrate version - 0.9.19 (as of the last time I tried this, and it still did not work).
Below is the error printed:
Migrate failed:  { [Error: Command failed: [ERROR] error: Column "migrations.id" has unsupported type "serial".
    at Connection.parseE (/Users/jclaus/Documents/Projects/Compass.Web/node_modules/db-migrate/node_modules/pg/lib/connection.js:534:11)
    at Connection.parseMessage (/Users/jclaus/Documents/Projects/Compass.Web/node_modules/db-migrate/node_modules/pg/lib/connection.js:361:17)
    at Socket.<anonymous> (/Users/jclaus/Documents/Projects/Compass.Web/node_modules/db-migrate/node_modules/pg/lib/connection.js:105:22)
    at Socket.emit (events.js:95:17)
    at Socket.<anonymous> (_stream_readable.js:765:14)
    at Socket.emit (events.js:92:17)
    at emitReadable_ (_stream_readable.js:427:10)
    at emitReadable (_stream_readable.js:423:5)
    at readableAddChunk (_stream_readable.js:166:9)
    at Socket.Readable.push (_stream_readable.js:128:10)
] killed: false, code: 1, signal: null }

Redshift does not support a serial column type, so ahead of time I tried creating the migrations table myself with correct column type (identity), and then hoping db-migrate could use it. Then I got this:
 ERROR:  [ERROR] error: relation "migrations" does not exist
    at Connection.parseE (/Users/Josh/Projects/Compass.Web/node_modules/db-migrate/node_modules/pg/lib/connection.js:534:11)
    at Connection.parseMessage (/Users/Josh/Projects/Compass.Web/node_modules/db-migrate/node_modules/pg/lib/connection.js:361:17)
    at Socket.<anonymous> (/Users/Josh/Projects/Compass.Web/node_modules/db-migrate/node_modules/pg/lib/connection.js:105:22)
    at Socket.emit (events.js:95:17)
    at Socket.<anonymous> (_stream_readable.js:765:14)
    at Socket.emit (events.js:92:17)
    at emitReadable_ (_stream_readable.js:427:10)
    at emitReadable (_stream_readable.js:423:5)
    at readableAddChunk (_stream_readable.js:166:9)
    at Socket.Readable.push (_stream_readable.js:128:10)

Ok, this is strange. I know from the issue #277 that migrations do work on redshift. This means something has changed somewhere. It's a bit tricky to figure this out, as I can't test it locally...
The second error, comes directly from the pg driver and not from db-migrate though. Seems like the table can't be found?
To the first error:
This is indeed strange, as you can see yourself from the linked issue, this has worked on redshift.
So we do need the serial type, which is used for the "autoincrement":
'id': { type: type.INTEGER, notNull: true, primaryKey: true, autoIncrement: true },

Just having a quick look at the docs:
http://docs.aws.amazon.com/redshift/latest/dg/r_CREATE_TABLE_NEW.html
Well ok, the pendant to autoIncrement for redshift seems to be IDENTITY. Uff, I hate it when developer split off from the original product that hard and dropping support of essential features... .
I see were we need to go though...
We need an additional driver, one that is basically one to one the pg driver, but with some minor changes. Like using IDENTITY for our auto incremented id column.
Actually I have a big problem with this:
I can't run this on our CI and thus can't test it properly and it doesn't seem like amazon would provide binaries...

@ice2morrow lets focus on the second error first. Can you somehow ensure that in the schema the migration table exists (with the identity instead of serial), maybe you created it in the wrong schema?
The error: relation "migrations" does not exist is mostly related to a missing table, or insufficent permissions.

@wzrdtales I can confirm the table is there, and with the correct column type when I create it manually. If I leave it to db-migrate, its created, but with the wrong column type, which throws the error.

@ice2morrow Then please do the following, get the queries generated by db-migrate through the verbose mode. And try executing them one by one manually and try to figure out where exactly the problem is.
As pointed out, this could also be a problem with missing permissions, which would cause a relation "" does not exist failure.
Can you also try version 0.9.17, with the migration table manually created. Just to make sure, there wasn't introduce a bug and we search at the wrong place...
Thank you

@ice2morrow Just to inforn you:
I will pick up this task for the redshift driver, which is going to use IDENTITY instead of SERIAL after I have finished on the linking between VC Seeders and Migrations.
Did you already had some time to try the queries manually, to discover where exactly everything stucks?

@wzrdtales I did what you asked, downgrading to 0.9.17 first, listed is the verbose output:
[INFO] Using dev settings: { driver: 'pg',
  database: '***',
  host: '***',
  port: '***',
  user: '***',
  password: '******',
  schema: 'dev' }
[INFO] require: ./pg
[INFO] connecting
[INFO] connected
[SQL] select version() as version
[SQL] SHOW search_path
[SQL] SET search_path TO "dev","$user"," public"
[SQL] SELECT table_name FROM information_schema.tables WHERE table_name = 'migrations' AND table_schema = 'dev'
[INFO] migration table created
[INFO] loading migrations from dir /Users/Josh/Projects/Compass.Web/database/migrations/warehouse
[INFO] loading migrations from database
[SQL] SELECT * FROM "migrations" ORDER BY run_on DESC, name DESC
[ERROR] error: relation "migrations" does not exist
    at Connection.parseE (/Users/Josh/Projects/Compass.Web/node_modules/db-migrate/node_modules/pg/lib/connection.js:534:11)
    at Connection.parseMessage (/Users/Josh/Projects/Compass.Web/node_modules/db-migrate/node_modules/pg/lib/connection.js:361:17)
    at Socket.<anonymous> (/Users/Josh/Projects/Compass.Web/node_modules/db-migrate/node_modules/pg/lib/connection.js:105:22)
    at Socket.emit (events.js:95:17)
    at Socket.<anonymous> (_stream_readable.js:765:14)
    at Socket.emit (events.js:92:17)
    at emitReadable_ (_stream_readable.js:427:10)
    at emitReadable (_stream_readable.js:423:5)
    at readableAddChunk (_stream_readable.js:166:9)
    at Socket.Readable.push (_stream_readable.js:128:10)

Then I ran them manually:
SHOW search_path;
$user, public
SET search_path TO "dev","$user", "public"
ran ok
SELECT table_name FROM information_schema.tables WHERE table_name = 'migrations' AND table_schema = 'dev'
came back with one row {table_name: 'migrations'}
SELECT * FROM "migrations" ORDER BY run_on DESC, name DESC
came back empty

The results were the same on both versions but below is the stacktrace from 0.9.19:
[INFO] require: ./pg
[INFO] connecting
[INFO] connected
[SQL] select version() as version
[SQL] SHOW search_path
[SQL] SET search_path TO "dev","$user"," public"
[SQL] SELECT table_name FROM information_schema.tables WHERE table_name = 'migrations' AND table_schema = 'dev'
[INFO] migration table created
[INFO] loading migrations from dir /Users/Josh/Projects/Compass.Web/database/migrations/warehouse
[INFO] loading migrations from database
[SQL] SELECT * FROM "migrations" ORDER BY run_on DESC, name DESC
[ERROR] error: relation "migrations" does not exist
    at Connection.parseE (/Users/Josh/Projects/Compass.Web/node_modules/db-migrate/node_modules/pg/lib/connection.js:534:11)
    at Connection.parseMessage (/Users/Josh/Projects/Compass.Web/node_modules/db-migrate/node_modules/pg/lib/connection.js:361:17)
    at Socket.<anonymous> (/Users/Josh/Projects/Compass.Web/node_modules/db-migrate/node_modules/pg/lib/connection.js:105:22)
    at Socket.emit (events.js:95:17)
    at Socket.<anonymous> (_stream_readable.js:765:14)
    at Socket.emit (events.js:92:17)
    at emitReadable_ (_stream_readable.js:427:10)
    at emitReadable (_stream_readable.js:423:5)
    at readableAddChunk (_stream_readable.js:166:9)
    at Socket.Readable.push (_stream_readable.js:128:10)

@ice2morrow Ok good, then this is nothing that was introduced with any of the last patches. Good to know!
So the node-pg driver (not the db-migrate pg driver) is returning the error at
 SELECT * FROM "migrations" ORDER BY run_on DESC, name DESC

Nothing new so far..., next question is:
Did you executed the queries with the same credentials you used in your database.json config for db-migrate? As the commands all ran fine while executing them manually, that could be also related to different permissions.
And concerning "came back empty", also try to insert some records into the migrations table.

@ice2morrow I think I have found the failure:
[SQL] SET search_path TO "dev","$user"," public"

Watching this cloooooosely, makes us aware of an space before public..
There was a pull request, for a similar bug that made it into the master concerning postrgresql 9.5 alhpa #282. Seems like the amazon devs are porting some buggy features. Going to release a patch for this.

Waiting for tag passing tests:
https://travis-ci.org/db-migrate/node-db-migrate/builds/75284843

Tag has passed the tests.
Released new patch v0.9.20.
Closed the issue, please try it with the newest version and reopen of necessary.

Added Issue concerning the issue with the SERIAL datatype.
@ice2morrow Did the released fix work for you?

@wzrdtales It looks like it did, thank you very much

Great, looking forward to create a better support for redshift, thus you and no one else have to create the migration table by hand :)


Plaintext agenda is only shown in UTC

Describe the issue
Usually the plaintext agenda was in the meeting time zone (or maybe multiple options were available?). I use the plaintext to print the agenda and would definitely prefer to have version that reflects the meeting time zone.
Code of Conduct

[X] I agree to follow the IETF's Code of Conduct

Duplicate of #5207. Will be fixed in the next release (which I hope to deploy today).


[Question] Customizing or translating "EditGrid header template

Hello,
Is there a way to translating the header template content (component.label) in edit grid.

       {% util.eachComponent(components, function(component) { %}         {% if (!component.hasOwnProperty('tableView') || component.tableView) { %}
{{ component.label }}
         {% } %}       {% }) %}

This is the default header template am seeing. but label text is not passed through translate function.

is there a way to call translate from here?
or is there an option to access array of components from this template "src/templates/bootstrap3/editgrid/html.ejs"; so that i can customize the editgrid template

@travist Any insights into this really helpful, Thank you.

This has probably been already fixed... Looking at header.ejs I can see the following default header template, which translates the header properly for me:
`

  {% ctx.util.eachComponent(ctx.components, function(component) { %}
    {% if (!component.hasOwnProperty('tableView') || component.tableView) { %}
      {{ ctx.t(component.label) }}
    {% } %}
  {% }) %}

`

Closing this thread as it is outdated. Please re-open if it is still relevant. Thank you for your contribution!


[3.4][SPARK-42826][FOLLOWUP][PS][DOCS] Update migration notes for pandas API on Spark.

What changes were proposed in this pull request?
This is follow-up for https://github.com/apache/spark/pull/40459 to fix the incorrect information and to elaborate more detailed changes.

We're not fully support the pandas 2.0.0, so the information "Pandas API on Spark follows for the pandas 2.0" is not correct.
We should list all the APIs that no longer support inplace parameter.

Why are the changes needed?
Correctness for migration notes.
Does this PR introduce any user-facing change?
No, only updating migration notes.
How was this patch tested?
The existing CI should pass

Merged to branch-3.4.


Feature request: Health globe color choice

Description
Change the color of the health globe from class specific to green
Version Number
latest
Steps to Reproduce
No bug (I think) just lack of Settings for the diabolicUI
Expected Behavior
Health Globe is green
Actual Behavior
Health Globe appears in class specific colours, no option to change
Additional Information
I'm missing other Settings too, for damage font etc
Error Log

I would say this is better as an option.
(Have the user choose a custom color)
In diablo the health orb is red, but I actually like it being the class color, but everyone has different tastes so I would suggest @goldpaw to do it as a customizable setting.

I consider it a requested feature, not a requested change. So we're all good. :)
I was thinking about implementing a semi-automatic system with not a fully user customizable choice, but a few predefined ones. Like diablo red, class color, warcraft'ish green, and so on. Something for everybody. :)

Maybe I worded the issue poorly, I wasn't requesting for it to be changed, but customization Options would be nice. Like the of course red (not green) orb. I'd use this UI all the time it wasn't for a few small things and its not even the healthglobes. Because I can't scale the UI the buttons in the taskbar are hard to see for example. Really just small issues. Other than that its really the best UI (especially Diablo one) I've seen.

You worded it perfectly, no worries! :)
And yes, a few small customizations like that are coming. I have actually built scaling into the UI, I just haven't gotten around to implementing the actual user menu for it yet. But it's coming!


Update sidebar.topHTML  after initialization

It seems there is a regression in the sidebar.
I cannot update sidebar.topHTML definition.
The code below already worked, and now it doesn't.
E.g. updating nodes still works...
(w2ui 2.0.x (nightly) (12/30/2023, 11:12:53 AM)
w2ui.sidebarRight['nodes'] = getSnapshotNodeDefinitions.value   // this works
w2ui.sidebarRight['topHTML'] = 'aaa'  // this doesn't work, the old content is still shown
w2ui.sidebarRight.refresh()

Are you sure it used to work? I checked the code and I see that it will update top and bottom html when render() method is called, but not referesh. Looking at the code, I see comments that say that topHTML should not be refreshed in refresh() as it might have input control with search for example, and then it would clear it. Refresh method updates nodes, I think it would be better to update topHTML and bottomHTML manually is needed after initial render

I propose we keep it like it is

I thing it woud be very usefull feature. E.g. you have sidebar with many items which number differs in time  and in the topHTML you have actuall number of items.

@jankrnavek , you can easily implement this feature directly updating html inside topHTML, I do think it should be done with grid.refresh(). Thank you.


Some datasets for languages.

I'm gonna practice drums for the rest of the day and probably won't work tomorrow, but for those who are looking to contribute and get some of those juicy points here is some low-hanging fruit in diverse languages:
Slovak:

Sentiment: https://huggingface.co/datasets/sepidmnorozy/Slovak_sentiment (as a matter of fact she has loads of Sentiment classification datasets: https://huggingface.co/sepidmnorozy)
News Summarization: https://huggingface.co/datasets/kiviki/SlovakSum

Greek:

Legal code clustering: https://huggingface.co/datasets/AI-team-UoA/greek_legal_code
NLI: https://huggingface.co/datasets/Harsit/xnli2.0_greek
Medical QA: https://huggingface.co/datasets/ilsp/medical_mcqa_greek

Maltese:

News NLI: https://huggingface.co/datasets/amitness/PAWS-X-maltese
News titles: https://huggingface.co/datasets/MLRS/maltese_news_headlines
News categories: https://huggingface.co/datasets/MLRS/maltese_news_categories

I'm gonna pick up kiviki/SlovakSum if noone is on it yet.

On the other hand it seems like the summary task requires:
        human_summaries: list[str]
        machine_summaries: list[str]
        relevance: list[float] (the score of the machine generated summaries)

and kiviki/SlovakSum doesn't have neither machine_summaries nor relevance scores.

@dokato Try formulating it as a retrieval task instead :))

I can start working on the Maltese dataset if no one is

I will take care of the sentiment classification dataset for Slovak, if no one claimed it yet

Will close this issue for now - I assume many of these are still relevant to add
@mariyahendriksen do you still want to add the greek medical QA?


646 Added support to parse appellate RSS Feeds

This PR adds support to query and parse appellate RSS Feeds.
Not too many differences between district and bankruptcy courts, the only ones are:

No pacer_case_id in appellate RSS feeds.
As in other sources, no document_number in appellate courts that don't have numbers yet.
Appellate RSS Feeds use local timezones for the date_filed, this is a difference from district/bankruptcy courts that uses GMT, so added support to parse properly date-times for all the US timezones.

Thanks, I've applied the changes and some comments above. I did an additional change that I realized while working on this from CL side, since there are still some courts that don't have document numbers, I added the same logic from AppellateDocketReport if the docket entry doesn't have a number the pacer_doc_id is assigned, so the entry is numbered and we can avoid duplicated docket entries.

Now using dateparser to parse date-times with timezones codes.
About Arizona, I checked RSS Feeds for courts that use US/Arizona like azd, and like other district courts it uses GMT, if at some point ARZ courts use a local timezone I think we'll get either PDT or MST that now we can parse, so seems we're now covered.

Great, thanks.


model search is broken

$ openplayground run
INFO:server.lib.sseserver:SUBSCRIBING TO: inferences
INFO:server.lib.sseserver:SUBSCRIBING TO: notifications
INFO:server.lib.sseserver:GETTING TOPIC: notifications
INFO:server.lib.sseserver:GETTING TOPIC: inferences
INFO:server.app:Initializing download manager...
INFO:server.app:Download loop started...
 * Serving Flask app 'server.app'
 * Debug mode: off
INFO:werkzeug:WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://localhost:5432
INFO:werkzeug:Press CTRL+C to quit
INFO:werkzeug:<IP_ADDRESS> - - [23/Apr/2023 00:02:20] "GET / HTTP/1.1" 200 -
INFO:werkzeug:<IP_ADDRESS> - - [23/Apr/2023 00:02:20] "GET /index.d826928d.js HTTP/1.1" 200 -
INFO:werkzeug:<IP_ADDRESS> - - [23/Apr/2023 00:02:20] "GET /index.e96bf64d.css HTTP/1.1" 200 -
INFO:server.lib.api:Getting enabled models
INFO:werkzeug:<IP_ADDRESS> - - [23/Apr/2023 00:02:20] "GET /api/models-enabled HTTP/1.1" 200 -
INFO:server.lib.api:Received notification request
INFO:server.lib.sseserver:LISTENING TO: notifications
INFO:server.lib.sseserver:LISTENING
INFO:werkzeug:<IP_ADDRESS> - - [23/Apr/2023 00:02:20] "GET /favicon.ico HTTP/1.1" 200 -
INFO:server.lib.api:Getting enabled models
INFO:werkzeug:<IP_ADDRESS> - - [23/Apr/2023 00:02:23] "GET /api/models-enabled HTTP/1.1" 200 -
INFO:server.lib.api:Getting enabled models
INFO:werkzeug:<IP_ADDRESS> - - [23/Apr/2023 00:02:29] "GET /api/models-enabled HTTP/1.1" 200 -
INFO:server.lib.api:Getting providers with models
INFO:werkzeug:<IP_ADDRESS> - - [23/Apr/2023 00:02:36] "GET /api/providers-with-key-and-models HTTP/1.1" 200 -
INFO:server.lib.api.provider:Searching Provider Models openplayground
ERROR:server.app:Exception on /api/provider/openplayground/models/search [GET]
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/lib/python3.10/site-packages/urllib3/util/connection.py", line 72, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 358, in connect
    self.sock = conn = self._new_conn()
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7fee4b826bc0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='openplayground.filler', port=443): Max retries exceeded with url: /api/search?q=gpt4 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fee4b826bc0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ettinger/.local/lib/python3.10/site-packages/flask/app.py", line 2528, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/ettinger/.local/lib/python3.10/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/ettinger/.local/lib/python3.10/site-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/ettinger/.local/lib/python3.10/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/ettinger/.local/lib/python3.10/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/home/ettinger/.local/lib/python3.10/site-packages/server/lib/api/provider.py", line 111, in provider_models_search
    response = requests.get(search_url)
  File "/usr/lib/python3.10/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 565, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='openplayground.filler', port=443): Max retries exceeded with url: /api/search?q=gpt4 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fee4b826bc0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
INFO:werkzeug:<IP_ADDRESS> - - [23/Apr/2023 00:02:57] "GET /api/provider/openplayground/models/search?query=gpt4 HTTP/1.1" 500 -
INFO:server.lib.api.provider:Searching Provider Models openplayground
ERROR:server.app:Exception on /api/provider/openplayground/models/search [GET]
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/lib/python3.10/site-packages/urllib3/util/connection.py", line 72, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 358, in connect
    self.sock = conn = self._new_conn()
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7fee4b824610>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='openplayground.filler', port=443): Max retries exceeded with url: /api/search?q=gpt4 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fee4b824610>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ettinger/.local/lib/python3.10/site-packages/flask/app.py", line 2528, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/ettinger/.local/lib/python3.10/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/ettinger/.local/lib/python3.10/site-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/ettinger/.local/lib/python3.10/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/ettinger/.local/lib/python3.10/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/home/ettinger/.local/lib/python3.10/site-packages/server/lib/api/provider.py", line 111, in provider_models_search
    response = requests.get(search_url)
  File "/usr/lib/python3.10/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 565, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='openplayground.filler', port=443): Max retries exceeded with url: /api/search?q=gpt4 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fee4b824610>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
INFO:werkzeug:<IP_ADDRESS> - - [23/Apr/2023 00:03:00] "GET /api/provider/openplayground/models/search?query=gpt4 HTTP/1.1" 500 -
INFO:server.lib.api:Getting enabled models
INFO:werkzeug:<IP_ADDRESS> - - [23/Apr/2023 00:03:11] "GET /api/models-enabled HTTP/1.1" 200 -
INFO:server.lib.api:Getting enabled models

I had to add huggingface api key


gem update w/ --prerelease ?

I wish to update to the latest commit.
I thought the following might work:
gem update yard  --prerelease
or
gem update yard-<IP_ADDRESS>  --prerelease
but it gives "nothing to update".
How do I get the latest commit into my gem directory ?

I ended up just d/l'g the ZIp and manually copying it, as I kept getting an invalid gemspec error trying to have gem.bat install it.

If you want to use the Git repository you should rely on Bundler and point your Gemfile at the repository directly. RubyGems don't map to Git commits.

Marking as closed since there is no way to install a RubyGem from Git source.

... which I found out when I tried to use the -s source param with the gem command.
It gave me some message like "no gem package found" or whatever.

@lsegal said:

If you want to use the Git repository you should rely on Bundler ...

Bundler is Cool! Didn't know about it.
gem install bundle went okay.
@lsegal said:

... and point your Gemfile at the repository directly.

Well it was your Gemfile. For any reader's info, I modified it to add the git source url to the "development" group, leaving the common gem source as the rubygems server.
From the top of the "GemFile" (in the root of the YARD project) :
source 'https://rubygems.org'

group :development do
  gem 'yard', :git => 'https://github.com/lsegal/yard.git'
  gem 'rspec', '= 2.12'
  gem 'rake'
  gem 'rvm-tester'
  gem 'simplecov'
  gem 'samus'
end

... and the rest of the "GemFile" continues ...
Switched to the the yard gem root. (Others who are collaborating on YARD might keep their projects else where.)
Tried bundle install:
... but then I had issues with Git only being setup while the GitHub Win client is running.
Had to read this: https://help.github.com/articles/set-up-git/
In order to configure Git Shell...
It creates a PowerShell script at %LocalAppData%/GitHub/shell.ps1
The paths are unique for every user, for security reasons.
I mimicked all the commands in it's powershell script, in a normal "cmd" batch script.
(My machine was configured to not run Powershell scripts for security purposes, and I had no desire to waste hours learning about MS Powershell. Never have found the time, nor had the desire.) Now I have a cmd script to run from my path, that sets environment vars and the %PATH% for using Git from the shell.
Tried bundle install again ... seemed like it was working.. then it choked trying to compile the  RedCloth gem.
Seems I had forgotten to configure the DevKit64 on the two latest Ruby installations...
... got that done (after relearning how to do that.)
Ie, switch to "C:/DevKt64"
ruby dk.rb init
ruby dk.rb install
The first time bundler choked installing RedCloth, it recommended (after properly configuring the Ruby Development Kit,) to install that alone as a test, ie:
gem install RedCloth-4.2.9
... which I did, and it went fine once Ruby 2.2 had access to DevKit64.
Finally the bundle install whilst in the gem folder seems to have worked:
C:\Ruby22-x64\lib\ruby\gems\2.2.0\gems\yard-master>bundle install
Updating https://github.com/lsegal/yard.git
Fetching gem metadata from https://rubygems.org/.......
Fetching version metadata from https://rubygems.org/..
Resolving dependencies...
Using rake 10.4.2
Using RedCloth 4.2.9
Installing asciidoctor 1.5.2
Installing diff-lcs 1.1.3
Installing docile 1.1.5
Installing locale 2.1.1
Installing text 1.3.1
Installing gettext 3.1.6
Installing json 1.8.2
Installing mob_spawner 1.0.0
Installing rack 1.6.1
Installing rspec-core 2.12.2
Installing rspec-expectations 2.12.1
Installing rspec-mocks 2.12.2
Installing rspec 2.12.0
Installing rvm-tester 1.1.0
Installing samus 1.4.3
Installing simplecov-html 0.10.0
Installing simplecov 0.10.0
Using yard <IP_ADDRESS> from https://github.com/lsegal/yard.git (at master)
Using bundler 1.9.9
Bundle complete! 12 Gemfile dependencies, 21 gems now installed.

Now, to write a cmd script to do this via a double-click. ...
P.S. This could added as a FAQ answer on the repo wiki, or where ever...


Reworking of the environment

WIP as of 12 March 2019

But you don't want to change recipe_spei.yml, right?

no, I just want to see what's going on on CircleCI, I'll do a merge of version2_development into this as soon as we are happy with the environment; at the moment it has iris=2.0.0 which is old as hell but at least other things work

OK I have a much cleaner and stable environment but I had to remove r-base that really, there is no way that can be a dep and not revert ncl to 6.4 which is horribly outdated (and it doesn't work at all) - @bouweandela any idea we can install that outside conda? Also note that cdo is at its latest, and now we are installing iris from PyPi

Good you are not reverting to NCL 6.4: we are using some functionalities which are only in NCL 6.5.
NCL 6.6.2 is not really needed.

OK @mattiarighi @bouweandela - I have managed to stabilise the environment:

cdo and ncl are free now (and conda installs them fine with their latest versions)
no more pinning on hdf5 (which now is the latest as well)
in the process I had a fling at installing iris from PyPi only but that has some kinks ie Cartopy fails to install from fresh (works the second time tho) and cf-units need to be pinned, pyke needs to be installed via conda -- in my conclusion iris is not ready yet to be installed from PyPi so it's back to conda for us
I had to add that gxx conda compiler in this configuration; no way we can do without it until conda fixes its stupid bug that the executable is bloody empty
MOST IMPORTANTLY I had to remove the r-base from the conda install; this package does not work at all with cdo and ncl current versions unless installed from the r channel but that's because the version there is old (3.2.2 as compared to the current 3.5.1) -- am looking now where to get it in our environment without any conflicts

OK I have tested extensively and found the two main conflicters with the r-base package: ncl and gtk2 - the annoying thing is gtk2 works with r-base for a certain combination of versions for each but that pair will not work with any ncl.  We need to install R in a different way other than conda or wait for conda to stabilize its trees for R=3.5.1 or higher. Here's my testing log if you would like to test as well -
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ vim environment.yml
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ conda env create -n esmvaltool_T4 -f environment.yml
Collecting package metadata: done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - gtk2==2.24.31=hde048ef_0
  - ncl
Use "conda search <package> --info" to see the dependencies for each package.

valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ vim environment.yml
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ conda env create -n esmvaltool_T4 -f environment.yml
Collecting package metadata: done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - gtk2==2.24.31=hde048ef_0
  - ncl=6.5 -> *[track_features=blas_openblas]
Use "conda search <package> --info" to see the dependencies for each package.

valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ vim environment.yml
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ vim environment.yml
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ conda env create -n esmvaltool_T4 -f environment.yml
Collecting package metadata: done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - gtk2=2.24.31
  - ncl
  - r-base
Use "conda search <package> --info" to see the dependencies for each package.

valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ vim environment.yml
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ conda env create -n esmvaltool_T4 -f environment.yml
Collecting package metadata: done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - gtk2==2.24.31=hb01d8f6_0
  - ncl
Use "conda search <package> --info" to see the dependencies for each package.

valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ vim environment.yml
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ conda env create -n esmvaltool_T4 -f environment.yml
Collecting package metadata: done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - ncl=6.5.0
  - r-base
Use "conda search <package> --info" to see the dependencies for each package.

valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ vim environment.yml
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ conda env create -n esmvaltool_T4 -f environment.yml
Collecting package metadata: done
Solving environment: failed

ResolvePackageNotFound:
  - r-base==3.5.1=h4fe35fd_1

valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ vim environment.yml
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ conda env create -n esmvaltool_T4 -f environment.yml
Collecting package metadata: done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - gtk2==2.24.31=hb01d8f6_0 -> glib=2.51
  - ncl=6.5.0 -> *[track_features=blas_openblas]
Use "conda search <package> --info" to see the dependencies for each package.

valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ vim environment.yml
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ conda env create -n esmvaltool_T4 -f environment.yml
Collecting package metadata: done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - ncl=6.5.0 -> *[track_features=blas_openblas]
  - r-base==3.5.1=he45234b_1005
Use "conda search <package> --info" to see the dependencies for each package.

valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ vim environment.yml
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ conda env create -n esmvaltool_T4 -f environment.yml
Collecting package metadata: done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - ncl
  - r-base==3.5.1=he45234b_1005
Use "conda search <package> --info" to see the dependencies for each package.

valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ vim environment.yml
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ conda env create -n esmvaltool_T4 -f environment.yml
Collecting package metadata: done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - ncl
  - r-base==3.5.1=h391c2eb_5
Use "conda search <package> --info" to see the dependencies for each package.

valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ vim environment.yml
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ conda env create -n esmvaltool_T4 -f environment.yml
Collecting package metadata: done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - ncl
  - r-base==3.5.1=h391c2eb_4
Use "conda search <package> --info" to see the dependencies for each package.

valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ vim environment.yml
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ conda env create -n esmvaltool_T4 -f environment.yml
Collecting package metadata: done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - ncl
  - r-base==3.5.1=he45234b_1004
Use "conda search <package> --info" to see the dependencies for each package.

valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ vim environment.yml
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ conda env create -n esmvaltool_T4 -f environment.yml
Collecting package metadata: done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - ncl=6.5.0 -> *[track_features=blas_openblas]
  - r-base==3.5.1=he45234b_1004
Use "conda search <package> --info" to see the dependencies for each package.

valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ vim environment.yml
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ conda env create -n esmvaltool_T4 -f environment.yml
Collecting package metadata: done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - gtk2==2.24.31=hb01d8f6_0 -> harfbuzz=1.3.4
  - gtk2==2.24.31=hb01d8f6_0 -> libtiff[version='>=4.0.3,<4.0.8']
  - r-base==3.5.1=he45234b_1004
Use "conda search <package> --info" to see the dependencies for each package.

valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ conda list harfbuzz
# packages in environment at /home/valeriu/anaconda3:
#
# Name                    Version                   Build  Channel
harfbuzz                  1.8.8                hffaf4a1_0
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ vim environment.yml
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ conda env create -n esmvaltool_T4 -f environment.yml
Collecting package metadata: done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - ncl=6.5.0 -> *[track_features=blas_openblas]
  - r-base==3.5.1=he45234b_1004
Use "conda search <package> --info" to see the dependencies for each package.

valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ vim environment.yml
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ conda env create -n esmvaltool_T4 -f environment.yml
Collecting package metadata: done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - gtk2==2.24.31=hde048ef_0
  - ncl
Use "conda search <package> --info" to see the dependencies for each package.

valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ vim environment.yml
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ conda env create -n esmvaltool_T4 -f environment.yml
Collecting package metadata: done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - gtk2==2.24.31=hde048ef_0
  - ncl
Use "conda search <package> --info" to see the dependencies for each package.

valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ vim environment.yml
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ conda env create -n esmvaltool_T4 -f environment.yml
Collecting package metadata: done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - ncl==6.5.0=blas_openblashd40de8d_1
Use "conda search <package> --info" to see the dependencies for each package.

valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ vim environment.yml
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ conda env create -n esmvaltool_T4 -f environment.yml
Collecting package metadata: done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - ncl==6.5.0=blas_openblash04324b8_2
Use "conda search <package> --info" to see the dependencies for each package.

valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ vim environment.yml
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ conda env create -n esmvaltool_T4 -f environment.yml
Collecting package metadata: - failed

CondaError: KeyboardInterrupt

valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ vim environment.yml
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ conda env create -n esmvaltool_T4 -f environment.yml
Collecting package metadata: done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - ncl==6.5.0=blas_openblash04324b8_3
Use "conda search <package> --info" to see the dependencies for each package.

valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ vim environment.yml
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ conda env create -n esmvaltool_T4 -f environment.yml
Collecting package metadata: done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - gtk2==2.24.31=hde048ef_0
  - ncl==6.5.0=blas_openblashf43dec5_4 -> *[track_features=blas_openblas]
Use "conda search <package> --info" to see the dependencies for each package.

valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ vim environment.yml
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ conda env create -n esmvaltool_T4 -f environment.yml
Collecting package metadata: done
Solving environment: failed

ResolvePackageNotFound:
  - ncl==6.5.0=blas_openblashf43dec5_0

valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ vim environment.yml
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ conda env create -n esmvaltool_T4 -f environment.yml
Collecting package metadata: done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - ncl==6.6.0=blas_openblashf43dec5_0 -> *[track_features=blas_openblas]
  - r-base==3.5.1=he45234b_1005
Use "conda search <package> --info" to see the dependencies for each package.

valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ vim environment.yml
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ conda env create -n esmvaltool_T4 -f environment.yml
Collecting package metadata: done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - gtk2==2.24.31=hde048ef_0
  - ncl[version='<6.6.0']
Use "conda search <package> --info" to see the dependencies for each package.

valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ vim environment.yml
valeriu@valeriu-PORTEGE-Z30-C:~/esmvaltool_alpha$ conda env create -n esmvaltool_T4 -f environment.yml
Collecting package metadata: done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - gtk2==2.24.31=hde048ef_0
  - ncl=6.4.0 -> *[track_features=blas_openblas]
Use "conda search <package> --info" to see the dependencies for each package.

Actually, was just thinking walking to the bus, it's probably high time we
split the environments - as I talked to @bouweandela at DLR, and he
suggested we break up the core from the diagnostics, we could start by
allowing for an NCL-based set of diags with its environment and an R,
Python etc.  I for one would
appreciate not going through the conda hell of the past few days again.
@mattiarighi go ahead and test the current env, it works fine (and I tested
on two different machines) so should be fine - apart from R, that is
Dr Valeriu Predoi.
Computational scientist
NCAS-CMS
University of Reading
Department of Meteorology
Reading RG6 6BB
United Kingdom
On Tue, 12 Mar 2019, 14:40 Mattia Righi<EMAIL_ADDRESS>wrote:

Good you are not reverting to NCL 6.4: we are using some functionalities
which are only in NCL 6.5.
NCL 6.6.2 is not really needed.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ESMValGroup/ESMValTool/pull/958#issuecomment-472027599,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AbpCoy9BFKj2IunPF_EftZ1X3YDzJU4Kks5vV7xngaJpZM4bqyLg
.

I agree that we should probably aim at a simpler environment just for the core functionalities (preprocessor and maybe the cmorizer, although these are also NCL).
I will test your last version today.

Cheers @mattiarighi
Dr Valeriu Predoi.
Computational scientist
NCAS-CMS
University of Reading
Department of Meteorology
Reading RG6 6BB
United Kingdom
On Wed, 13 Mar 2019, 07:40 Mattia Righi<EMAIL_ADDRESS>wrote:

I agree that we should probably aim at a simpler environment just for the
core functionalities (preprocessor and maybe the cmorizer, although these
are also NCL).
I will test your last version today.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ESMValGroup/ESMValTool/pull/958#issuecomment-472313136,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AbpCoxJiN3Jeoi6SRz7S2gCYGOp49EUkks5vWKt6gaJpZM4bqyLg
.

Thanks @valeriupredoi, it works at DKRZ: no errors from ncl nor cdo!

Awesome! Cheers @mattiarighi ! @bouweandela - any ideas about how to
integrate that r-base without breaking the environment (for now, before we
start thinking about separating envs) - I can't see any way with conda (at
least the current tree does not allow for it unless r-base is very old
which will not support the installation of all the other extra R packages).
It is not on PyPi either so that probably leaves us either an apt-get
install or we can include it in the esmvaltool source and build it straight
from there?
Dr Valeriu Predoi.
Computational scientist
NCAS-CMS
University of Reading
Department of Meteorology
Reading RG6 6BB
United Kingdom
On Wed, 13 Mar 2019, 08:22 Mattia Righi<EMAIL_ADDRESS>wrote:

Thanks @valeriupredoi https://github.com/valeriupredoi, it works at
DKRZ: no errors from ncl nor cdo!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ESMValGroup/ESMValTool/pull/958#issuecomment-472324146,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AbpCo62OpGA0v1PgAxcWk0Re251Ra9c0ks5vWLU4gaJpZM4bqyLg
.

ENVIRONMENT WORKS WITH ALL THE NEEDED CRAP IN IT!!! :beers:
have a look at https://github.com/NCAR/ncl/issues/87 - that Keving guy is a life saver :grin:
I created the env from scratch, installed the tool and ran the tests; cdo and ncl are slightly older but who cares (cdo=1.9.5 and ncl=6.5.0); bloody conda better fix this in the future, I think this frozen mirror will work for now, question is when will it become obsolete, but at the moment I will need a beer and forget about this

related to https://github.com/NCAR/ncl/issues/87 and Kevin's comments, we really need to investigate what the hell is conda up to with that compiler and things will be fixed beyond the cf201901 frozen mirror

Thanks for all the work on this. I just tried the new environment and ran into an error. Since it might be relevant, I post it here.
crezees@exo:~/projects/env_ymls> conda env create -f 20190314_version2_development_solve_environment_March2019.yml -n esmvaltool-solve_env
Collecting package metadata: done
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - gxx_linux-64
Use "conda search <package> --info" to see the dependencies for each package.

Couldn't get much more info on the conflict.

Have you run this with the latest env file (from last evening, with the
cf201901 label conda forge channel)? If so, pls remove the gxx compiler
from the dependency list in environment.yml, that should probably not be
there for that frozen environment
Dr Valeriu Predoi.
Computational scientist
NCAS-CMS
University of Reading
Department of Meteorology
Reading RG6 6BB
United Kingdom
On Thu, 14 Mar 2019, 08:57 bascrezee<EMAIL_ADDRESS>wrote:

Thanks for all the work on this. I just tried the new environment and ran
into an error. Since it might be relevant, I post it here.
crezees@exo:~/projects/env_ymls> conda env create -f 20190314_version2_development_solve_environment_March2019.yml -n esmvaltool-solve_env
Collecting package metadata: done
Solving environment: failed
UnsatisfiableError: The following specifications were found to be in conflict:

gxx_linux-64
Use "conda search  --info" to see the dependencies for each package.

Couldn't get much more info on the conflict.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ESMValGroup/ESMValTool/pull/958#issuecomment-472759940,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AbpCo5kz1OYe-BIep1hSS4dwS_5lNBdWks5vWg7ugaJpZM4bqyLg
.

Yes, I used the very latest version. I removed it and it works fine now, also went through the installation smoothly. Thanks!

Cheers for testing! I'll remove it on the branch too, that damned conda
compiler is the root cause of all our (and many others') conda-related
woes, really don't get it why conda folk need to complicate things
unnecessarily.
Dr Valeriu Predoi.
Computational scientist
NCAS-CMS
University of Reading
Department of Meteorology
Reading RG6 6BB
United Kingdom
On Thu, 14 Mar 2019, 09:04 bascrezee<EMAIL_ADDRESS>wrote:

Yes, I used the very latest version. I removed it and it works fine now,
also went through the installation smoothly. Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ESMValGroup/ESMValTool/pull/958#issuecomment-472762417,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AbpCo0HN2a7HM9F7wDrEvq2AM8bPb-HWks5vWhChgaJpZM4bqyLg
.

Hi V, great work figuring stuff out! I've created a clean branch which implements your solution here version2_environment. Maybe we can use that for a pull request instead?

Sure, man, good call! can you pls add me to that PR, I'd like to add a
comment on the gxx compiler and a comment poiting to the actual decision of
conda guys to create that frozen mirror, for our internal records. Cheers!
Dr Valeriu Predoi.
Computational scientist
NCAS-CMS
University of Reading
Department of Meteorology
Reading RG6 6BB
United Kingdom
On Thu, 14 Mar 2019, 10:00 Bouwe Andela<EMAIL_ADDRESS>wrote:

Hi V, great work figuring stuff out! I've created a clean branch which
implements your solution here version2_environment
https://github.com/ESMValGroup/ESMValTool/tree/version2_environment.
Maybe we can use that for a pull request instead?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ESMValGroup/ESMValTool/pull/958#issuecomment-472782244,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AbpCo8MCTK2Y_b4I_SoJ1iNkmAGjfdZaks5vWh3BgaJpZM4bqyLg
.

Replace by #965


Use checked arithmetic for addition to Unix epoch.

The current code can panic with filesystems that do not store times. Using checked arithmetic allows such filesystems to operate without panic.

I'm not sure this is the right fix. That code path is only taken when the appropriate FATTR_CRTIME...etc flag is set. Do you know why that flag is getting set without a valid timestamp passed in?

Hello, I'm having an issue that would be solved by this PR.
I've done some investigation and I think the issue lies on macFuse side: macfuse/macfuse#1042
@cberner, could this PR be merged ? Or does it require additional work ?

Going to close this in favor of https://github.com/cberner/fuser/pull/322


Linux kernel type library

When analyzing Linux kernel modules, it would be great if we have the type libraries, so that code like this becomes more readable:

Most importantly, it helps us know what these callback functions do:

checker.ko.zip
https://app.hackthebox.com/challenges/SEPC

We've discussed this internally and it's about an impossible task. We certainly couldn't ship any type libraries, as the Linux Kernel provides zero guarantees of compatibility. The only solution would be to generate them on behalf of our customers given a k-config and kernel source.


SecurityError: The operation is insecure.

https://jsfiddle.net/tcbwm7L1/
I assume the issue is that input[type=file] elements cannot be recycled, or at least Firefox 53.0.2 doesn't like it.

https://jsfiddle.net/tcbwm7L1/2/
Object notation works.

there are a several workarounds to this. in the end they all boil down to differently keying the vnodes to prevent them from being recycled: https://jsfiddle.net/tcbwm7L1/1/
a little-know fact (that will be documented soon) is that in the absence of a _key, vnodes will be auto-keyed by _ref, _id, or name [1].

Object notation works.

wow, that's quite surprising and worth further investigation.
[1] https://github.com/leeoniya/domvm/blob/2.x-dev/src/view/initElementNode.js#L42-L49

the problem here is order of assignment. if you try to set the value before the type is changed, it'll screw over.
with object notation it breaks too: https://jsfiddle.net/tcbwm7L1/3/
it's a side-effect of how selector-style attrs are merged into any provided attrs object. if explicit attrs exist, then that object is simply augmented with the additional attrs. i don't see a cheap way of avoiding this situation since it will still occur if you swap the order and define el("input[value=#eee]", {type: "file"}).
i think documenting that input/type recycling should be avoided by keying/id-ing/naming/_ref-ing inputs.

the problem here is order of assignment. if you try to set the value before the type is changed, it'll screw over.

So is there any argument to prioritize certain attributes in patchAttrs, or is type the only edge case?

So is there any argument to prioritize certain attributes in patchAttrs, or is type the only edge case?

the roadblock is perf. patchAttrs is a very hot function, even adding a single condition to it will have a very noticeable impact on perf across the board, even when type is not present.
the entire attr patcher is a simple for-in loop over an merged attrs object. since there is no way to add hash keys at the beginning of an existing object, it's not something that can be done cheaply.
i'm sure there are other edge cases, but they will be equally difficult solve for the same reason. the universal solution is to just avoid recycling in each case - it doesn't even make sense to recycle certain types of inputs (text->number probably fine, radio->number probably not fine).
it's an unfortunate choice W3C made to have all input types with wildly different UIs and use-cases to be grouped under a single input tag.

For what it is worth, my real use-case is much more complex than the boiled down jsfiddle. In the real code, it is not at all obvious that the input node would be recycled.
I'd rather not develop the habit of adding _key by default, but I guess #135 would suffice with a warning when a collision does happen.

I'd rather not develop the habit of adding _key by default,

it may not be terrible to get in the habit of adding a semantic input[name=...] in by default, though. this input-reuse case is the precise reason i added the auto-keying behavior for attrs that could reasonably be expected to be unique among sibling nodes.
it's a difficult/expensive problem to solve at runtime in non-compiled [and not statically analyzable] libs.

Haha, I literally just finished adding name to all my inputs. I'm satisfied!


[RFC] Remove usage of popper.js's API

https://github.com/mui-org/material-ui-pickers/blob/b62ce2a71aa1395f02b7d54e11dbbfab8a080ea6/lib/src/_shared/PickerPopper.tsx#L70-L71
we rely on the API of popper.js's (onCreate), however, this leads to 3 issues:

It forces us to write async tests which should almost never be needed.

https://github.com/mui-org/material-ui-pickers/blob/d6134e05c71361f1a6c2d1ea210690e04d544c83/lib/src/__tests__/DateTimePickerTestingLib.test.tsx#L44
Without the wait for, it fails with:

It's unclear if this API will stay around in the future https://github.com/mui-org/material-ui/issues/19358
It could make it harder to expose the API to extern users.

I think that we could consider using a ref, to know when the DOM is safe to be usable

It seems it was already handled


App Only Routes

Currently if you want to access all the topic, or create new ones all you need to do is read the code base and run some REST calls against the /topic endpoints in the server/init/routes.js file.
By default routes like this should only be accessible by the application rather than available to anyone who looked at this one file.
Trying to think of a good way to do this... possibly some kind of Application key? Thought of trying to use a check source location but each client would be different... plus if any idiot reads through the compiled source They would see a client side API key... what about tying it to the passport?  What would that look like?
Pros - Prevents non-public API calls via exposed routes for application.
Cons - Another layer of complexity.

This can be achieved by require a anti csrf token being passed with each request. Something like:

On server, pass a csrf token to the client
On client for subsequential API requests, pass the token
On server, validate the token

The token (cookie) can be made more secure, with some settings such as same-origin

We're trying to clean up the project and close issues.
The solution that was posted is very viable and good. Basically you should have authentication implemented.
In case you still have an issue with this, please comment here or open a new issue.

Question:
In a fork of this repo, I have a very simple csrf express middleware which uses the express csruf middleware, that would be easy to add to this repo.
Usage
app.use('/api', csrfProtectionMiddleware, apiMiddleware)

Is there any interest in a PR adding this?

You would also need to change the front-end to include the csrf token on the requests.
It can add a layer of security and it will help but it does require a bit more working on it.
It is probably recommended to add CORS protection to verify the origin of the request. When would you decide to refresh the csrf token?
Also, it we go to the path of JWT then we might not need the csrf token. Check this and this

I actually added this to my app with out using CSRF.  By adding the following middleware to my routes.  This allows me to be selective on which routes have the login check versus the blanket method the CSRF token uses.
server/init/routes.js
function ensureLoggedIn(req, res, next) {
  /*
   * This checks is the isAuthenticated function exists and it's value
   * If the request came from a user (browser) with an active session the route will execute
   * If the request came from the server with a server authorization header it will also work
  */
  if ((req.isAuthenticated && req.isAuthenticated()) ||
    req.headers.server_authorization === sessionSecret) {
    next();
  } else {
    res.status(401).send({
      success: false,
      message: 'You need to be authenticated to access this page!'
    });
  }
}
// Example
app.post('/my-list', ensureLoggedIn, myController.list);

The tricky part was getting it to work with the middleware.  If you refresh the page the server side axios calls in the fetch data promises don't contain the user's session which is why I added the server_authorization header.  As long as this head is set and unset it'll never been seen on any client just on the requests where the server is talking to itself so we don't need to reuse our code.
update "match" portion in server/render/middleware.js
match({routes, location: req.url}, (err, redirect, props) => {
    if (err) {
      res.status(500).json(err);
    } else if (redirect) {
      res.redirect(302, redirect.pathname + redirect.search);
    } else if (props) {
      // This method waits for all promises to resolve before returning to browser
      store.dispatch({ type: types.FETCH_DATA_REQUEST });

      // Intercept the axios requests and add a session secret to the header.
      const myInterceptor = axios.interceptors.request.use((c) => {
        const config = c;
        config.headers.common.Server_Authorization = sessionSecret;
        return config;
      });

      fetchDataForRoute(store.dispatch, props)
      .then((response) => {
        response.forEach((promiseResponse) => {
          // console.log(promiseResponse);
          // In event of error data fetcher should return appropriate error response.
          store.dispatch({ type: promiseResponse.type, data: promiseResponse.data});
        });
        const html = pageRenderer(store, props, req);
        res.status(200).send(html);
      })
      .catch((error) => {
        console.error(error);
        res.status(500).json(error);
      });
      // Remove the axios interceptor
      axios.interceptors.request.eject(myInterceptor);
    } else {
      res.sendStatus(404);
    }
  });

I may still add anti CSRF at some point but for now this is what I needed.

great :)
I believe that you can simplify this and use the regular Cookie header on the SSR side instead of a custom header name. Check https://github.com/reactGo/reactGo/pull/937

@slavab89 Thanks again this is what I was trying to do but could figure out how to pass the cookie to the axios call from server side.  Thanks for knowing this boilerplate inside and out ;)


ECC: uncorrectable error: ffffffffffffffff ff when flashing fake GUARD without erasing.

[root@ltc-xxxx ~]# pflash  -p fake.gard  -P GUARD
About to program "fake.gard" at 0x0002c000..0x0002c03f !
WARNING ! This will modify your HOST flash chip content !
Enter "yes" to confirm:yes
Programming & Verifying...
[=                                                 ] 0%
Updating actual size in partition header...
[root@ltc-xxx ~]# opal-gard list -all
ECC: uncorrectable error: ffffffffffffffff ff
 ID       | Error    | Type       | Path
ECC: uncorrectable error: ffffffffffffffff ff
-----------------------------------------------------------
 00000001 | 90000002 | Fatal      | /Sys0/Node0/Membuf5
ECC: uncorrectable error: ffffffffffffffff ff
===========================================================
libflash ecc invalid
[root@ltc-xxxx ~]#

Yes, programming flash without erasing first will do these kinds of things. What is your expected behaviour here?

@cyrilbur-ibm As we have #151 causing the problems to erase and flash, i just tried flashing without erasing to see whether that works or not :)

not a bug


Fixed consistency between unw/concomp/coh indices

Ensures consistency between product indices in stack VRT files. Addresses issue raised in #241 .
Thanks @yunjunz and @totaibi01 for tracking this down!

Thanks @sssangha @dbekaert for solving this out
Thamer


allow config and port types to be undefined

which represents when COGS has not yet connected

I think we should change output port values as follows:

Defaults must be passed into constructor (cannot be undefined)
Therefore this.currentOutputPortValues cannot be undefined

This is actually pretty challenging to write with Typescript and may even cause more confusion. What I'm looking for:

The constructor should support no output ports, in which case it makes sense omit the argument altogether.
The constructor should force you to enter the output ports in the case the type parameter to CogsConnection contains outputPorts.

Closing in favour of version 2+.


Node Erorr

Hello everyone. Tell me what is the problem?
 To enable parallel build, add the "/ m" parameter.
unpack_xz_dep
nothing.c
win_delay_load_hook.cc
nothing.vcxproj -> C: \ dexcalibur \ node_modules \ xz \ build \ Release \ nothing.lib
configure_and_make
"eval" is not internal or external
command, executable program, or batch file.
C: \ Program Files (x86) \ MSBuild \ Microsoft.Cpp \ v4.0 \ V140 \ Microsoft.CppCommon.targets (171.5): error MSB6006: "cmd.exe" ended with code 1.   Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR!  C: \ Users \ User \ AppData \ Roaming \ npm-cache \ _logs \ 2020-12-15T18_08_14_232Z-debug.log
NodeJs version 12.0.0 64 bit. Python 2.7 64bit. Registered in a system variable.
Tried so
npm install --global --production windows-build-tools --vs2015
npm config set msvs_version 2015 –global
npm config set python python2.7
npm install
The error remains.

Node-gyp v3 vs. the current v7

See https://github.com/nodejs/node-gyp/issues/2272 for steps to upgrade node-gyp (not node).


Add batch size flag to examples

Summary of Changes

Add a batch flag to the examples
Sync C++ and Python examples
Print when starting a local server

Closes #92
Motivation
These quality-of-life changes improve the examples.
Implementation
The batch flag is added for both C++ and Python. The two versions of the examples are also synced with each other.

Build failed!

Passed internally


added CreateThreadCommandTest

Pull Request Test Coverage Report for Build 285

11 of 11 (100.0%)  changed or added relevant lines in 3 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+1.4%) to 53.294%

Totals

Change from base Build 282:
1.4%

Covered Lines:
1626

Relevant Lines:
3051

💛  - Coveralls


Export aggregated PUMS

Export Finished PUMS Indicators (demographics, household economic security) to Digital Ocean
New workflow: Export PUMS Aggregated
This iterates through 3 geographies (puma, borough and citywide), 2 years (2012 and 2019) and 2 categories (demographics and household economic security). It calls one new .py file external_review/external_review_PUMS and a new case in the existing export_DO.sh  file. Everything else is re-used from existing workflows.
It usually takes around 2 hours to run with all data.
External Review PUMS
This file is responsible for calling the new load_aggregated_PUMS and passing it variables from a Typer command line interface. It gets a dataframe of collated results for a single category and saves it to the .staging folder.
Load Aggregated
This was written to be similar to load_data.py in the ingestion folder. Similarly to how that file can deliver PUMS or HVS, this was written to deliver PUMS indicators and hopefully new functions can be written to deliver other ones as well. It has logic to look for a cached file and run the aggregations if no cached file is found.
Reorder and rename columns
All these changes are in aggregate/PUMS/aggregate_PUMS.py
Columns are reordered so that each category for each indicator has columns for it's counts and fractions grouped together, and then crosstabs on that category follow after each category is listed. Ages and income run from low to high, education runs from more educated to less educated. Other categories (industry, occupation) are listed alphabetically.
"Fraction" is changed to "pct" and variance abbreviations (cv, moe) are sent to lowercase.
Cache aggregated results
This upgrade doesn't effect the export directly but was necessary to test the external review code on my local machine. Aggregated data are saved to .output folder. There is no "requery" parameter like there is for the ingestion process, that's a possible upgrade. To re-run results just deleted cached .csv
Cache filenames are created in in aggregate/aggregated_cache_fn.py
While I was going through the cache process for raw PUMS to replicate it's structure I cleaned out some unnecessary parameters.
Remove Medians
All medians calculations we have are incorrect, they should be based on buckets. They are taken out of external review upload and exceptions are raised if the aggregation is attempted.
Miscellaneous Changes
Other workflows were modified to only be run when it's corresponding .yml is changed. This is because exports don't need to be run on push and we aren't getting helpful info back from test workflows
Tests were changed to conform to the new column name specifications. They weren't tested as some as out of date anyway
Notes for future work
This utilizes two new gitignored folders, data/ and .staging/. There may be simpler way to store these temporary files, possible place for refactor.
Additionally the existing external review and new load aggregated data processes both collating multiple indicators into a single category for export. This is poor design and should be refactored to be made consistent

Note that github action is failing with error code 137. https://github.com/NYCPlanning/db-equitable-development-tool/runs/5307165703?check_suite_focus=true
Google suggest it's a memory issue, trying with test data only to see if that works

This fails for economics, need to figure out a fix. Think it's a memory issue


Make beaker available as a package through snaps

Make it possible to download and install a packaged version of beaker through snapd, really the best and easiest way to make a program available on all linux repositorys 👍

Cool we'll target this next. Should have an appimage install for the next release too

really the best and easiest way to make a program available on all linux distributions

@Jacalz did you try the Beaker AppImage? Just download it, set the executable bit, and double-click it.

I know that is an option but no I did not try that because snap is way simpler to just run a command and boom you are done, executable files is not really my cup of tee 😅

+1
Also, snaps are (or could be, it demands more of the app, not sure how beaker would fare here) sandboxed by design. Check out build.snapcraft.io for free, automated builds.

Hey @pfrazee I work on the Snapcraft team and would love to help get Beaker published in the Snap Store, which is the app store for Linux :smiley:
I've just tested the snap you're already producing as it looks awesome! Snapcraft now automatically clears the executable stack, so the potential blocker discussed above is resolved.
I see you're using Travis and they offer snap publishing support, see their docs:

https://docs.travis-ci.com/user/deployment/snaps/

Do you need any more help to get Beaker published in the snap store? Once published, you'll have direct control of the publishing life cycle and get access to metrics for weekly active devices. We'd love to feature Beaker as an editors pick and put together a social campaign to promote it!


Issue 1807

i18n of language restart dialog fixes #1807

Both eslint and jshint are generating warnings for many files.  Should I raise another issue and start resolving them by fixing the code?
I've always beleived that unless all warnings are fixed in the code base, contributors will be lazy.

We added eslint and set rules for it after a lot of the code base already existed, so while fixing warnings is appreciated, it's not a requirement.
Eslint should always be used instead of jshint.

I've rebased to the master

I had to rebase to master because of conflicts.

It looks like something went funny with the rebase, it's trying to re-apply changes that have already been made. I'll take a look tomorrow and see if I can get it synced properly.


Interview Preparation details

Description 📜

Hello! I am GWOC'21 Participant and I want to add the folder for Interview Preparation. I will add all the details about how to prepare for the interviews and from where. Can you please assign this issue to me?

Domain of Contribution 📊

[ ] Android Dev (Flutter, Java, Kotlin)
[ ] Backend Dev (Java, .NET, Python)
[ ] C/CPP
[ ] Competitive Programming
[ ] Cyber Security
[ ] DSA
[ ] Database
[ ] Datascience with Python
[ ] Datascience with R
[ ] Frontend Dev (HTML, CSS, JS)
[ ] Frontend Dev (React, Angular, Vue)
[ ] Golang
[x] Interview Prep
[ ] Java
[ ] JavaScript
[ ] MERN
[ ] Machine Learning
[ ] Open Source
[ ] Python
[ ] Rust
[ ] Statistics
[ ] UI/UX

/assign


Kinesis service controller

New ACK Service Controller
Support for Amazon Kinesis
List of API resources
List the API resources in order of importance to you:

Stream

Keep in mind that ACK only handles control plane operations, not data plane operations, so Kinesis API operations like SubscribeToStream, DeregisterStreamConsumer, PutRecord and PutRecords are not going to be supported.
Some Kinesis API operations that are going to be problematic:

IncreaseStreamRetentionPeriod
DecreaseStreamRetentionPeriod
EnableEnhancedMonitoring
DisableEnhancedMonitoring
StartStreamEncryption
StopStreamEncryption
UpdateShardCount

Because they represent mutating operations on just a single attribute of a stream and there isn't a ModifyStream or UpdateStream API call, which means we're going to need to inform the code generator in ACK what all those API operations actually do...

MergeShards
ListStreamConsumers

Neither of these is congruent with the Kubernetes API and declarative modeling.
So, at least initially, it's only going to be CreateStream and DeleteStream and then followed by the "single attribute mutating operations".

Thanks @jaypipes pointed out operational focus. Just want to confirm will embedded inform code into generator to handle these kind of service or not determine yet?

@cc4i we will be enhancing the code generator in ACK to account for these kinds of odd API calls, yes. Already starting to do that with the Elasticache and RDS ACK service controllers, since both of those APIs have a lot of rough edges like this :)

/lifecycle frozen

@cc4i and all other watchers: you can install and use the Kinesis controller for ACK now.
You can install it using the kinesis-chart Helm chart.
Feedback and criticism very much welcomed!
-jay


{AKS} Vendor new SDK and bump API version to 2022-11-02-preview

This checklist is used to make sure that common guidelines for a pull request are followed.

{AKS} Vendor new SDK and bump API version to 2022-11-02-preview.
Fix test case test_aks_nodepool_add_with_disable_windows_outbound_nat.  Set the ossku of the new nodepool to Windows2019, since Windows2022 does not support this feature at present.

For new extensions:

[ ] My extension description/summary conforms to the Extension Summary Guidelines.

About Extension Publish
There is a pipeline to automatically build, upload and publish extension wheels.
Once your pull request is merged into main branch, a new pull request will be created to update src/index.json automatically.
The precondition is to put your code inside this repository and upgrade the version in the pull request but do not modify src/index.json.

AKS

@FumingZhang Do you need to upgrade a new version to release this PR?

Hey @zhoxing-ms, I think it's not necessary to release a new version. It would be fine to release a new version after some new features are included.


Add a reentrancy module

This PR:

adds a new module reentrancy.sw
exposes is_reentrant() & reentrancy_guard() functions
moves helper functions to call_frames.sw
adds better documentation for const values
adds tests

PR discussion history: https://github.com/FuelLabs/sway-lib-std/pull/34

I'm adding at least 1 more test and reviewing the logic more closely before requesting reviews here.

Hmm. One failing test... it's making we question my assumptions about how this is working. As far as I can see, the reentrancy::can_call_guarded_function  test should be passing.

Ok, after digging into this for a while, I have a few notes:

all tests are failing. (2 show passing as they are expected to revert, but they're reverting for the wrong reason). I will update tests to capture this.
all tests failing with ContractCallError("MemoryOverflow")
while looking hard at this, I did find a bug in my implementation for is_reentrant().
TL:DR: I previously had this line updating the  call_frame_pointer value:
call_frame_pointer = get_previous_frame_pointer(call_frame_pointer);
after the id comparison if previous_contract_id == this_id {...
which effectively was causing the function to always return true. (because previous_contract_id would then always be the same as this_id for the first iteration).

Re: the test logic
Definitely a bit convoluted!
TL;DR;

In all cases but 1, the SDK calls the Attacker contract, which calls the Target contract, which calls back into the Attacker contract, which calls back into (reenters) the Target contract before the first call to the Target contract has finished executing.
The exception to this pattern is the can_call_guarded_function() test, which just demonstrates that a function guarded by the reentrancy_guard() function can still be successfully called (as long as the call is not reentrant).

The other tests show that we can:

detect reentrancy and act on it in any way we want
strictly forbid reentrant calls
detect reentrancy across functions:

Technically, our method of detecting reentrancy operates at the contract level, not at the function level. So, if Attacker contract, calls  Target::foo() , which calls back into the Attacker contract, which calls back into Target contract (but this time calls  Attacker::bar()), is_reentrant() inside of Target::bar() will return true.
This is not ideal, but I think it's ok. This does not prevent ContractA from making multiple successive calls into ContractB, only from doing so in a reentrant manner. I can't think of a good reason to do the latter, but would love feedback on this!

cc @adlerjohn @otrho

@otrho All tests are passing in the current state.
The errors mentioned resulted when I tried to implement suggested changes (adding an explicit return true;).


#203 Use EncryptedSharedPreferences instead of raw preferences

This PR enables the usage of EncryptedSharedPreferences instead of the plaintext sharedpreferences.
Note that this increases the minimum api-level from 18 to 23.

I think it's a good feature however it's not ok to just raise the min SDK because of this. We should implement a function that checks if api-level is lower or equal to/higher than 23  then we should use EncryptedSharedPreferences, otherwise fall back to the existing function. Also, it should be an opt-in because people would loose their existing data if we just migrate everyone to EncryptedSharedPreferences.@nilsmagnus do you have any time to implement this?

Closing this in favor of #259


export statement in hooks/index

The index file at 'packages/nextjs/hooks/scaffold-eth/index.ts' does not export all of the defined hooks. Three export statements need to be added:
export * from "./useAccountBalance";
export * from "./useAnimationConfig";
export * from "./useDeployedContractNames";

@resjudicoda Thanks!


chore: release version (#57)

This pull request releases a new version.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 97.078%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
299

Relevant Lines:
308

💛  - Coveralls


Dependencies still showing out of date for gulp-protractor

Hello,
https://david-dm.org/mllrsohn/gulp-protractor/
The dependencies is shown out of date even though details show everything up to date.
Any clue what is missing from my end.
https://david-dm.org/rahulmr/gulp-protractor/ shows everything green

same,
I am getting this with:
https://david-dm.org/dolanmiu/docx
All of it is green, why is the badge red, its so puzzling

Same issue here:
https://david-dm.org/Cloud-CV/cvfy-frontend?type=dev

Same issue:
https://david-dm.org/theoephraim/node-pg-migrate

Same issue here https://david-dm.org/mattstratton/linter-cookstyle

Same issue here https://david-dm.org/dawson-org/dawson-cli :(

I have found a solution.
Use this instead. It is much more reliable and accurate:
https://gemnasium.com/github.com/dolanmiu/docx

This project seems to be not maintained anymore.

@tocttou this project IS maintained. If you'd like to submit a pull request to fix the issue, it would be appreciated.

Oh cool, I'll take a look.

@alanshaw Is the project really maintained?
I mean, two month of a problem with one of the core features, the badges, and it seems like nothing will happen here?! I think the main reason to use david are the badges, wich seems not to work correctly anymore, related to the amount of topics about the badges here. And the wrong informations the badges commit will cause users to not use some project, because it seems insecure, like your badge is telling them.
There is even no information from you on any topic here, the david page or the blog I've found. So no information about problems, no answer for two month and no updates for three month.
But you call it still maintained?

Thanks for your patients. I believe this bug has been resolved in by https://github.com/alanshaw/david-www/commit/6436f07bec18384d0d83a8e89b7fb40286da5bc8. Please comment if you notice this issue again.

@alanshaw seems fixed but on devDependencies I see it is orange when it should be red as one of the dev dependency is out of date and red. Check here

It shows yellow when < 25% of dependencies are out of date


DroppedAsset reports invalid clickType

Issue tracker is ONLY used for reporting bugs.
Summary
DroppedAssetFactory → droppedAsset → updateClickType()
I’m giving clickType: "webhook” as a parameter but it’s giving me error:
{
    "errors": [
        {
            "message": "Invalid value",
            "location": "body",
            "field": "clickType"
        }
    ]
}

documentation here is not specific: https://metaversecloud-com.github.io/mc-sdk-js/enums/types.DroppedAssetClickType.html#LINK
Summarize the changes that have been made to the platform.

Expected Behavior
Tell us what should happen.

Current Behavior
Tell us what happens instead of the expected behavior.
I’m giving clickType: "webhook” as a parameter but it’s giving me error:
{
    "errors": [
        {
            "message": "Invalid value",
            "location": "body",
            "field": "clickType"
        }
    ]
}

Possible Solution
Not obligatory, but suggest a fix/reason for the bug.

Steps to Reproduce
Provide a link to a live example, or an unambiguous set of steps to reproduce this bug. Include code to reproduce, if relevant.

Step one...

Context (Environment)
Providing context helps us come up with a solution that is most useful in the real world.
How has this issue affected you? What are you trying to accomplish?

Detailed Description
Provide a detailed description of the change or addition you are proposing.

Possible Implementation
Not obligatory, but suggest an idea for implementing addition or change.

This is not a problem with the SDK but rather a feature enhancement needed in the Public API to allow adding webhooks to assets. I've added this to our roadmap.


remove all refs to naked HOME or '~' from README

by 'naked HOME or '~'' I mean '~/' alone.  '~/.ssh' is fine as it
clearly states what directory out of the home directory is needed.
This is being done to clarify what directories are needed if a user
doesn't want to use the standard directories
this fixes #96

I prefer https://github.com/samsung-cnct/k2/pull/182/files#diff-04c6e90faac2675aa89e2176d2eec7d8

Plus, my version more closely emulates what's actually being done in k2cli.

I forgot you had done this already.  I like your version better as well.


VAN backbone performance?

VAN-Small + Light-Ham-D256 with 15.8GFlops and 13.8M Params achieves 45.2mIoU on ADE20K: here
MSCAN-S  + Light-Ham-D512 with 16.0GFlops and 14.0M Params achieves 44.3mIoU on ADE20K
VAN-Base + Light-Ham with 34.4GFlops and 27.4M Params achieves 49.6mIoU on ADE20K: here
MSCAN-B + Light-Ham with 35.0GFlops and 28.0M Params achieves 48.5mIoU on ADE20K
What is the need of MSCAN backbone? The paper explain that "Though VAN has achieved great performance in image classification, it neglects the role of multi-scale feature aggregation during the network design, which is crucial for segmentation-like tasks", however that's vanilla VAN without Light-Ham decoder

For Ham repo, it reports the multi-scale and flip testing result.
For fair comparison, it should compare 45.2 v.s. 45.8 n 49.6 v.s. 49.9.
And for more models like Tiny and Large, SegNeXt also has a improvements.
If we test the single scale inference result, we can find SegNeXt has a more visible progress.

Thank you!
So MSCAN can leverage multi-scale feature through InceptionNet-like block while VAN can't right?

Yes.
VAN is a strong backbone for various vision task.I think VAN is the proudest job I have done so far.
About the differences between VAN and SegNeXt:

From MSCAN view, it introduces multi-scale information, which is important for dense prediction task.
Besides, strip decomposition of large kernel convolution seems more friendly to classification and segmentation tasks.

From SegNeXt, it not only contains an encoder, but also introduces how to conduct a suitable decoder in detail.  We have made a lot of attempts in the decoder to make SegNeXt effective and lightweight including the struture and algorithm (Figure 3 in SegNeXt).  After some attempts, we choose current decoder, as you can see in the SegNeXt and Ham repo.

Both LKA and MSCA have a 3D attention map like Residual Attention .

Both LKA and MSCA have a 3D attention map like Residual Attention .

Thank you for your fast response. I have never seen this idea before studying about Attention in CNN. I'll take a look into it!


If fixed header is set, it's duplicated and columns don't respect the specified width

Issue Type

[x] Bug

Specs
What version are you using?
2.20.0
What browser?
Chrome
Expected Behavior
What did you expect to happen?

Only one row header should appear
And it should be synced with the table content when scrolling to the right
The header should respect the width set for each column

Actual Behavior
What actually happened?

Duplicated header rows appear
The columns of the fixed header doesn't have the specified width for columns
Fixed header doesn't scroll laterally with the table content (move to the right of the rows)

Steps to Reproduce the Problem
Please detail your steps here

Set :fixed-header="true"
Specify max-height="150px", i.e. a value in which the number of rows per page (10) doesn't fit in that height
Add more rows (+8) to make the effect appear (see Fiddle)

jsfiddle
This fiddle reproduces the bug:
https://jsfiddle.net/brjapon/sgnko39w/12/

duplicate of https://github.com/xaksis/vue-good-table/issues/697

have the same issue and the resolved ticket does not contain the solution.
on browser refresh I get a duplicated header, navigating in and out of the page via routes the fixed header is fine


Follow up to #188: Shorten the Contact Us section

#188 was merged (by me) believing that the HVNC had already vetted the text. Commentary on that merge suggests otherwise. From the last comments on that PR:
@ifokkema wrote:

This also adds a reference to a method to ask us a question on GitHub:

You may also submit questions to our
GitHub repository Q&A board.

Is it a good idea to offer multiple methods for this? Is one preferred over the other? Does the deadline that we set for ourselves also count for the GitHub method of asking us a question?

@marinadistefano wrote:

Yeah, I do have concerns about missing something that it on our GitHub Q&A board instead of via email.

@reece wrote:

If it's still up for discussion, then 1) I agree that we should drop the Q&A board, 2) I think the text about the mailing list usage, turn around time, etc. is far too long. (Preview at https://hgvs-nomenclature.org/main/).
I recommend:

merging the first two paragraphs ("Join the..." and "Messages sent to...").
dropping the entire "When members of the community..." paragraph
dropping the Q&A board paragraph

Also, in a separate PR, I proposed to rename this section and add a new announcement-only group. See https://hgvs-nomenclature--191.org.readthedocs.build/191/.

Since that PR is merged, let's @ahwagner @marinadistefano @ifokkema continue the conversation here.

Yes, there was a desire to be more transparent in the way the we respond to communications with the community so I drafted that text. I do think it's important that the community know how long it will generally take us to respond as many contact us to seek advice on actual clinical cases.

Sorry for any confusion here; I am responsible for adding the Q&A board text. I thought it was an obvious enhancement since we already provide that as an alternate community forum for GitHub-familiar people to ask and answer questions. This was added during my efforts to restructure this section to incorporate the text that @marinadistefano put together. I considered this a minor change and so only requested approval from @marinadistefano (as the task lead) to view the restructured section and text integration in #188. I am okay with removing the Q&A board text since this is apparently unpopular, and similarly fine with merging the first two paragraphs. However, I do not think we should remove the HVNC-vetted text that @marinadistefano contributed without committee review.


Disconnection event spam when servers are not set

[03:36:47 ERROR]: Couldn't pass DisconnectEvent to leaf
java.lang.NullPointerException: Cannot invoke "com.velocitypowered.api.proxy.server.RegisteredServer.getServerInfo()" because "server" is null
        at com.github.smuddgge.leaf.datatype.User.addHistory(User.java:191) ~[?:?]
        at com.github.smuddgge.leaf.events.EventManager.onPlayerLeave(EventManager.java:56) ~[?:?]
        at com.github.smuddgge.leaf.Leaf.onPlayerLeave(Leaf.java:108) ~[?:?]
        at com.github.smuddgge.leaf.Lmbda$14.execute(Unknown Source) ~[?:?]
        at com.velocitypowered.proxy.event.UntargetedEventHandler$VoidHandler.lambda$buildHandler$0(UntargetedEventHandler.java:47) ~[server.jar:3.1.2-SNAPSHOT (git-8deb0ff7-b198)]
        at com.velocitypowered.proxy.event.VelocityEventManager.fire(VelocityEventManager.java:598) ~[server.jar:3.1.2-SNAPSHOT (git-8deb0ff7-b198)]
        at com.velocitypowered.proxy.event.VelocityEventManager.lambda$fire$5(VelocityEventManager.java:479) ~[server.jar:3.1.2-SNAPSHOT (git-8deb0ff7-b198)]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]

This error is sent whenever a server isn't specified when someone disconnects (unconfigured server)

Hey, what version are you using currently? I believe I fixed this issue in the latest version :3


Update BouncyCastle from 1.69 to 1.78.1

Motivation:
The update was held back by a compatibility.
Since in 4.2 we've moved the JDK baseline from Java 6 to Java 8 anyway, we have the opportunity to bump the BC version as well.
Modification:
Switch all BouncyCastle dependencies from "-jdk15on" to "-jdk18on", and update them to the latest version.
Result:
Fixes #13561

See their release notes for all the changes: https://www.bouncycastle.org/releasenotes.html#r1rv78

Revapi is correctly pointing out that this is a breaking change. Will add overrides in a bit.

@chrisvest thanks a lot !


Harvested crawls not displayed

I'm currently setting up WebCurator and have a test environment ready.
Thanks to your great tutorial, I've set up the first crawls. After some time, if I come back to dashboard, it reports 4 target instances "ready for review":

However, if I click on "harvested", only the first one is displayed:

I checked the WCT Store directory, and there are all the crawls I started:
root@webarchive:/usr/local/wct/store# find . -name *.warc ./12/1/IAH-20220328150009474-00000-813~webarchive.net.adk.de~8443.warc ./33/1/IAH-20220325105013639-00000-158564~webarchive.net.adk.de~8443.warc ./29/1/IAH-20220325103842200-00000-158564~webarchive.net.adk.de~8443.warc ./37/1/IAH-20220325135817400-00000-158564~webarchive.net.adk.de~8443.warc ./26/1/IAH-20220330125535725-00000-813~webarchive.net.adk.de~8443.warc ./28/1/IAH-20220325103517901-00000-158564~webarchive.net.adk.de~8443.warc ./31/1/IAH-20220330154537316-00000-813~webarchive.net.adk.de~8443.warc ./35/1/IAH-20220330164504871-00000-813~webarchive.net.adk.de~8443.warc
If I enter another crawl ID manually in the search form, the respective crawl is displayed (url is clickable and shows details about the crawl):

I would expect all finished crawls to be displayed in the table. What am I doing wrong here?
Thanks,
Torsten

The page doesn't seem to get rendered properly, html just stops at
28/03/2022 17:00:01
Harvested
T. Roloff
00:01:08:10
54,22 MB
3795

HTML source ends here.
I found something in the logs:
2022-03-31 13:33:40.297 +0200 ERROR [http-nio-8080-exec-86] o.a.j.l.DirectJDKLog (DirectJDKLog.java:175) - Servlet.service() for servlet [dispatcherServlet] in context with path [/wct] threw exception [Request processing failed; nested exception is org.apache.tiles.request.render.CannotRenderException: JSPException including path '/jsp/qa-queue.jsp'.] with root cause java.lang.NumberFormatException: For input string: "0,0"
Seems to be an i18n issue? I'll look into /jsp/qa-queue.jsp to see if I can find the value causing trouble.

Ok, managed to fix it on my own:
I had a locale de_DE set on my server, and the wct-webapp service was started with this locale. Now I'm passing "LANG=en_US" before starting the service, and everything seems to be ok.
Problem was wrong locale when testing a number (line 896 of jsp/qa-queue.jsp):
<c:when test="${intFailed < 100.0}">         <------- this fails when wct-webapp service is started with "wrong" locale
Strange enough, as far as I understand locale is set  on line 739 with
<fmt:setLocale value="en_GB" />
but seems to be ignored?