New Famine dataset

This is the new Famine dataset compatible with v21 routing product attributes.
Mohammad

Looks good. All I would suggest it is to move the zip into a folder. i.e.: raven-testdata/famine/HRU_Famine_v21.zip. Feel free to merge after that!

Looks good. All I would suggest it is to move the zip into a folder. i.e.: raven-testdata/famine/HRU_Famine_v21.zip. Feel free to merge after that!
Thanks @Zeitsperre ! but @cjauvin said the zip folder should not be within another folder...

Thanks @Zeitsperre ! but @cjauvin said the zip folder should not be within another folder

Sorry for the confusion. I just meant that the shapefile content (i.e. the .shp, cpg, etc files) should not be in a folder, INSIDE the zip.


Configuration option for startLLM.py outout format

Feature request
Ability to set the output to [Sources, Question, Answer] instead of [Question, Answer, Sources].
Motivation
In my use, it is easier to see the generated response if it is always at the bottom of my terminal, since I rarely want to look at the actual sources that were used.
Your contribution
I would be willing to do a PR if this sounds like an OK idea and I get a bit of guidance. On the other hand, if you agree that the default should be [Sources, Question, Answer], then it is a really easy change.

At the current rate I can handle and review PRs so feel free to open up one with a brief description.
Sounds reasonable too.


feat: add optional useRepaintBoundary to Animate

This is a draft PR
Documentation is still open, I will add it next week when I have more time.
closes #46

@gskinner I added a first draft of the docs, let me know what you think! And feel free to change anyting obviously.
As a side note: The dart docs for parameters are only available with the "standalone" widgets, not with the extension methods. I can totally see not wanting to copy/paste the docs due to various reasons.
A possible way of circumventing this would be to use dartdoc Macros as described here.
What do you think about that?


Webpack Error when npm run start with the current version extension

Hi team,
I got a following error when npm run start in Looker Extension Framework Setup step.
xxx@xxxx:~/looker-explore-assistant/explore-assistant-extension$ npm run start

><EMAIL_ADDRESS>start
> webpack serve --hot --server-type https  --port 8080 --config webpack.develop.js

[webpack-cli] You need to install 'webpack-dev-server' for running 'webpack serve'.
Error: Cannot find module 'webpack-dev-server'
Require stack:
-<EMAIL_ADDRESS>- /usr/share/nodejs/webpack-cli/lib/webpack-cli.js
- /usr/share/nodejs/webpack-cli/lib/bootstrap.js
- /usr/share/nodejs/webpack-cli/bin/cli.js
- /usr/share/nodejs/webpack/bin/webpack.js

When I used the previous commit version of c986fe36f173a13490f7458a8252e7a63db22a1a, it worked well without the same issue in my same dev environment.

@lucidhwee have you run the npm run install  command to install all dependencies?
refer this document at this step: https://medium.com/refined-and-refactored/how-to-set-up-genai-x-looker-extension-under-15-minutes-explore-assistant-b1f918ed9c58#:~:text=run this command-,npm install,-Next%2C we will
PS: please note this document is not updated as per the latest changes. there is one addition task to be done to setup completely i.e. inserting sample examples in bigquery. please refer PR https://github.com/looker-open-source/looker-explore-assistant/pull/75 for this change.


Inconsistent timezone updates

Currently the calendar updates based on the timezone in which the schedule is viewed (https://www.chilconference.org/calendar.html#tab-calendar). However the other tabs, e.g. Thursday and Friday do not.
Low priority, but has created some confusion with times overlapping. Easy fix might be to indicate that all times are ET.

@mghassem @tnaumann should we just remove https://www.chilconference.org/calendar.html#tab-calendar ?
It's just one additional thing to keep updated, and I'm not sure that it adds any value.

I would. :l
On Fri, Jul 10, 2020, 11:08 AM Tom Pollard<EMAIL_ADDRESS>wrote:

@mghassem https://github.com/mghassem @tnaumann
https://github.com/tnaumann should we just remove
https://www.chilconference.org/calendar.html#tab-calendar ?
It's just one additional thing to keep updated, and I'm not sure that it
adds any value.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/acm-chil/acm-chil-website/issues/22#issuecomment-656811351,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAV5XDAJAOV5RDFWXGBCC53R25KLNANCNFSM4OWYARBQ
.

Okay, the calendar tab is no more!: https://www.chilconference.org/calendar.html.
@tnaumann does this address the issue?

Yes, this resolves the issue.


ability to hide/unhide sidebar

[ ] do hide/unhide (switch classes col-9 to col-12 for container)
[ ] hide link/button, could be certainly inside (top) of sidebar
[ ] hide/unhide button  could be a small little unhide button on top of notebook (right-side), or could also live in somewhere in the menu (but don't fit well)
[ ] less important: also don't like the sidebar to use js/onscroll magic. it should be a position: fixed, but then we need to attach it somewhere elsewhere in the DOM...

cc @andypetrella

yeah, agreed the position thingie would be better but then I guess we won't be able to use the grid system (that's probably your comment about its position in the DOM right?)
:+1:

huh, not sure we need the grid at all.
maybe tabs would be better? (doesn't matter now, this will evolve naturally with new functionality...)

sure, the grid isn't necessary, I was more thinking about the refactoring.
Also, I'm wondering if have a real grid system like gridster that would also manage the main notebook would be interesting (then we can arrange at our will).
Anyway, show/hide is :+1:
Also, we'd probably need a menu for tuning the panels we'd like to show out-there (and an api to add/register new ones)

like gridster that would also manage the main notebook would be interesting (then we can arrange at our will).

maybe better think about contextual menus around each cell for "up" & down buttons.
also multi-column arrangement makes no sense until we have no doc/title per graph/per cell. not yet sure, but maybe it would make sense to allow to attach markdown directly to a cell?


Python bindings for enry

Same way as we have Java bindings for enry, wrapping a Go library built with  -buildmode=c-shared it would be nice to have one for Python using ~ctypes FFI~ cffi or something similar.
Particular use case: one want to use https://github.com/bblfsh/sonar-checks/ API that would require knowing a language the file-to-be-checked is written in, to choose the write checks.

ctypes is an ancient deprecated way, cool kids use cffi.

This is just nice-to-have: for now, the use case listed above is addressed in lookout by incorporating language in DataServer API https://github.com/src-d/lookout/pull/131

I've started adding it during OSD and will push simple WIP with CPython extension by cffi soon

Issue description updated to include the scope.


Adding cypress data attributes

Summary
Add cypress data attributes for fast element finding in tables

Occurred changes and/or fixed issues

Technical notes summary

Areas or cases that should be tested

Areas which could experience regressions

Screenshot/Video

A bit out of scope, but may you include where you want to use the attribute for creating E2E tests in the PR description?

A bit out of scope, but may you include where you want to use the attribute for creating E2E tests in the PR description?

When we create a pod through e2e test, we need to go back to the list page to find out if the pod we created is shown in the list, we need to locate the location of the pod by namespace and name,   and we also need to check if the other columns of the pod are displayed as expected.

Done.  PTAL， thanks.

Thank you for the clarification 🙇


[backport/1.23] repo: Improve key signing of binaries (#23640)

Signed-off-by: Ryan Northey<EMAIL_ADDRESS>
Commit Message:
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]

/retest


chore(deps): bump go to v1.15

Looks like the repo did not stay with the latest go, thus bumping it to v1.15

Redoing this #160. Thanks for the reminder @chenrui333 ☺


Active JJDoc and let it create the Grammar BNF documentation

This will generate the BNF documentation on the Maven Site:

I still will need to create a sensible Stylesheet for it and maybe write a Sanitizer for the HTML page and add Railway diagrams.

Coverage remained the same at 88.502% when pulling 1b5d133f7387b51d4913062db5c6f6b601325e97 on manticore-projects:ActiveJJDoc into 111244ab565a160f050a2812b372a5f0db5255e3 on JSQLParser:master.


Feature import export

Added a button to the bottom left of the Settings page for Import/Export Settings:
Text is auto-selected when they open the dialog and settings are stored in storage after user clicks "Apply"

I love this one! I'll do a full review tonight.

Perfectly done, thank you! The only other thing I'd recommend is representing this on a Trello card so others know it's happened. 👍


Browser compatibility

@shrayshray is getting errors when trying to view the staging site in IE11 and Firefox68.

The Firefox error looks like the CSS/JS files aren't loading correctly. The IE error is more mysterious, but when testing with BrowserStack I get this:

Investigation in progress.

For IE11:
The SCRIPT1006 error is triggered by this line of code:
https://github.com/datamade/la-metro-councilmatic/blob/8462d7467d2c0cfa1d223f7f1a3a5f13c7d39c07/lametro/static/js/autocomplete.js#L2
According to this, IE doesn't support default arguments in functions. We'll need to re-write this function for the script to work (or have a chance of working) in IE.


Base64 with newlines

Not strictly a bug, but it would be very convenient to be able to paste a block of text in base64 and have your app  filter out any newline characters. Thanks - J

Whitespaces (including new line characters) will now be removed before decoding base64 / ascii85. Thank's for your feedback!


Choosing workspace-enabled extensions explicitly in extensions.json

I use many different extensions to program in many different languages.
The problem is that some conflict with others.
For instance, if I would have Paredit enabled all the time, then, some crucial features of plain text editing become unavailable (i. e. ctrl+right to cursorWordEndRight).
This is partly solved by disabling extensions and then workspace-enabling it, but I suppose it would be even nicer to just specify enabled extensions per workspace (like, in .vscode/extensions.json)
I hope you'll like this idea. I personally find it very convenient.

Duplicate of #40239


Avoid redefining 32-bit integer types like UINT32_TYPE

Describe the bug
Zephyr redefines built-in 32 bit integer types. While the arm-none-eabi-gcc package shipped by ARM uses "long int", Zephyr uses just "int". The end result may look like the same in C, but this creates problems when linking external C++ libraries. C++ mangles types into the symbol name. "long int" and "int" result in different mangled symbol names. When linking a Zephyr application to an externally built C++ library, symbols will not be found.
Expected behavior
I expect Zephyr to keep all types as defined by the toolchain.
Impact
Linker errors when linking external C++ libraries built with the same toolchain.

This was a deliberate decision as described in the issue you linked (https://github.com/zephyrproject-rtos/zephyr/pull/16645).
For now, the expectation for the third-party libraries to be used with Zephyr is that they should include the Zephyr zephyr_stdint.h globally so that they use the same type system as the upstream Zephyr code.
The plan is to eventually get rid of this "hack":
https://github.com/zephyrproject-rtos/zephyr/issues/37718#issuecomment-917435777

Please add something like the following when you are building third-party libraries:
https://github.com/zephyrproject-rtos/zephyr/blob/bf3cd116e9f553c33b7d2ef483a3ea00132a5bf1/CMakeLists.txt#L288

Thank you for the suggested workaround. I will instead define a replacement type for uint32_t in the public API of our library.
I would be glad if this hack is removed.

Closing in favour of https://github.com/zephyrproject-rtos/zephyr/issues/46032, which provides the implementation plan for this.


Only allow connected accounts to view DAO

Closes #157
This UX manifests itself in two scenarios:

When attempting to load a DAO URL directly, and a user doesn't have their wallet connected
When viewing a DAO, and then the user manually disconnects their wallet

I could seriously use some extra eyes on this from @da-colon @tbwebb22 @sethhrbek

I could seriously use some extra eyes on this from @Da-Colon @tbwebb22 @sethhrbek. Let me know if you fellas are available for a screen share on Wednesday

After discussing with the fellas, came up with a solution that involved tracking when an account was being loaded, which is different than the two states of "connected account" and "no connected account".
So implementation-wise, force navigate back home if we're done "loading the account" and there is no account there.

This is ready for review :pray:


Please release the updated library

After merging this PR the library runs fine on iOS Simulator. But the library cannot be easily pulled in any react-native application package.json directly from main like so
    "react-native-fast-tflite": "mrousavy/react-native-fast-tflite#main",

because the tensorflow submodule appears empty in node_modules/react-native-fast-tflite/tensorflow.
Thats why I kindly ask @mrousavy to release the updated version of the library (1.3.1). For the users to be able to use it in their react-native projects without suffering the inability to run on iOS Simulator. Thank you in advance!

released 1.4.0 👍

@mrousavy doesn't seem to compile now, any one else is having the same issue?
https://github.com/mrousavy/react-native-fast-tflite/issues/91

Doesn't work on Android in 1.4.0 ! But it works on IOS 😭

Here is the error:
node_modules/react-native-fast-tflite/android/../cpp/TensorflowPlugin.h:19:10: fatal error: 'tensorflow/lite/c/c_api.h' file not found
#include <tensorflow/lite/c/c_api.h>
@mrousavy

Same issue here ! Anyone knows how to fix it without cloning all the tensorflow repository into the react native tflite fast module each time ?

@manu13008 have you tried pointing to master in instead of the npm version?
Ex of package.json:  "react-native-fast-tflite": "git://github.com/mrousavy/react-native-fast-tflite#master",

Okay. So nothing worked for me on Windows :( I still get the same error as previous

1.4.0 builds on iOS simulator, but I am seeing #91 errors in Android


GraphQL - Support for JS pipeline resolves

Is this feature request related to a new or existing Amplify category?
api
Is this related to another service?
AppSync
Describe the feature you'd like to request
At present, Amplify only supports extending the Amplify-generated resolvers for GraphQL APIs using VTL.
AppSync now supports writing pipeline resolvers in JS.
Describe the solution you'd like
Support for extending Amplify-generated resolvers using JS.
Describe alternatives you've considered
Writing custom pipeline resolvers in VTL is of course possible, but generally makes for a less pleasant developer experience (harder to test, less familiar syntax).
Business logic can also be encapsulated within a Lambda function resolver. However, when the required logic is relatively simple (e.g. performing some extra validation before a Create/Update operation), this feels like overkill.
Additional context
No response
Is this something that you'd be interested in working on?

[ ] 👋 I may be able to implement this feature request

Would this feature include a breaking change?

[ ] ⚠ This feature might incur a breaking change

Hey @james-kuihi :wave: thanks for raising this! As @auvred mentioned this is a duplicate of #1015. Closing in favor of tracking the existing item


Created a form for create product in prod page

feat(product): Created a form for create product in prod page

refactor


Maps should be able to specify owner users/characters

At the moment if a map is opened up to a corp/alliance, anyone can expand the permissions without it being tracked or immediately obvious -- this is bad for opsec.
You should be able to specify owners and have it so only owners can change map permissions.

Can't you control scope with the the pathfinder.ini file?  Meaning if you only allow corp access (by putting a corp into the ini) and then you open the map to that corporation .. who would they be expanding the scope to?

When someone creates a map of privacy type 'Corporation' then goes to the last tab, you can specify the corporations that have access to that map. By default it is only the users corporation but you can add more; that means that anyone in the original corporation can add an entire second corporation to the map.
Even worse they could delete the map or set it to private and remove access for everyone but themselves.
In pathfinder.ini you can limit the maximum number of corporations that can be added to one map so you could in theory set this to 1, but there are legitimate use cases where adding more than 1 corp is completely valid. You can also restrict login to one corp but the same problem applies.

; restrict login to specific corporations/alliances by id (e.g. 1000166,1000080)
CORPORATION                 =
ALLIANCE                    =
You put the ID of the corp in there, and only those corps can log in

Yes but you may want an entire alliance to have access to Pathfinder but still want to allow corporations within that alliance to have their own private maps. If Corp A and Corp B are members of Alliance A and Pathfinder is limited to Alliance A in the login restrictions, someone from Corp A could add Corp B to Corp A's map, even though you don't want them to have access.
Even if you took the overly restrictive approach and had one pathfinder per corporation or something silly, it still means that in a 10,000 person corporation -- all it takes is one spy to delete the entire corporations map. You should be able to restrict map deletion/permission changes to a small group of people.

ah, got it.  Carry on then.  :-)

Yeah, would be nice but requires some kind of "role" management.

Who is allowed to set map invites, who is not?

If a map is shared between corporations/alliances there is a "share icon" next to the map tab. This should help to identify shared maps. But seriously, if someone already has access to a map and want to share those information with an enemy. There is nothing you can do against that. - Share screenshots is always an option...

It is but this is just as much about protecting from accidental deletion as
much as opsec. For example we had a user think that the settings permission
boxes were for them to set what Corp they were in -- you'll always have
dumb users but it would be nice to raise the bar a little bit in the
technology.
I was thinking a second box in the sharing settings where you can just list
additional characters that have 'owner' permissions in the map.
On Monday, May 23, 2016, Mark Friedrich<EMAIL_ADDRESS>wrote:

Yeah, would be nice but requires some kind of "role" management.

Who is allowed to set map invites, who is not?

If a map is shared between corporations/alliances there is a "share icon"
next to the map tab. This should help to identify shared maps. But
seriously, if someone already has access to a map and want to share those
information with an enemy. There is nothing you can do against that. -
Share screenshots is always an option...
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/exodus4d/pathfinder/issues/164#issuecomment-221085186

Ok, sounds good!

When you create a map for the first time, you will be the only one who can add other owners.
Everyone who is added added as "owner" can invite other owners and edit share settings.

There is still a problem:
Everyone in a corporation is able to create corporation maps as well. In case there is a spy, he can create a corporation map and invite others. If we bind "map delete" option to owners as well, no one will be able to delete this map. I think we need some kind of "corp/ally" leaders with "admin" rights who always have "owner permission" for a corporation/ally map. But there is no way to automatically get "managers" from CREST.

I agree it would be nice if for a corporation we could have people
pre-identified as the owners for that corporation but that does seem like
something better solved with a more extensive role framework.
I would be inclined toward accepting that we could end up with some Corp
maps littered and inaccessible for the moment and look toward implementing
something more extensive in the future.
Curious on other peoples thoughs on this though!
On Monday, May 23, 2016, Mark Friedrich<EMAIL_ADDRESS>wrote:

Ok, sounds good!

When you create a map for the first time, you will be the only one
who can add other owners.
Everyone who is added added as "owner" can invite other owners and
edit share settings.

There is still a problem:
Everyone in a corporation is able to create corporation maps as well. In
case there is a spy, he can create a corporation map and invite others. If
we bind "map delete" option to owners as well, no one will be able to
delete this map. I think we need some kind of "corp/ally" leaders with
"admin" rights who always have "owner permission" for a corporation/ally
map. But there is no way to automatically get "managers" from CREST.
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/exodus4d/pathfinder/issues/164#issuecomment-221087915

I´ll think about an acceptable solution for this problem :)
For self-hosting installations we could think about a configuration form within the /setup path, where you can add "admin" characters for corporations.
But this will not work on "shared-installations" like mine. There are currently 253 alliances and 948 corporations with registered users.

Agreed. I was thinking more about this and the permissions systems in general -- I'm wondering if it would be better for maps not to be private/corp/alliance but just to be 'private' and the settings window allow you to specify which of the ones you want. I can see situations where people want 1 alliance, 2 non-corporation alliances and 1 character not in any corp or alliance to have access to a map.
Random side thought anyway -- I'll keep thinking on the ownership front.

I'm planning on forking this repo and putting some time into addressing the permissions issue.
I like the idea of map owners and map based permissions. Map permissions include: permission/role assignment, edit map, add to map by jumping, view map, view player locations on map.
I think also users should have ownership of their location data and be able to choose which maps to share that data with. You're then trusting the map owner to share that data with the right people so individuals don't have to micromanage this themselves.

+1

Can you please confirm if you are able to correct the ability to access and share out corp maps with other groups via the search field. At this time the function of the map owner sharing out the map to groups will no populate the corp names if searching via atleast 3 characters in version 1.1.7.

I can confirm that searching for characters to add to a private map does not work, even if the character searched for is the original owner (after having removed the owner, of course). My guess would be a parsing error from reading EVE's entity data.

@Judahmeek have you read the manual? the character you are searching for needs to activate "map sharing" before he can be found by others. This should prevent map invite spamming.

What is new in v1.3.3 ?

Any character needs to re-auth their chars and accept the new esi-corporations.read_corporation_membership ESI scope , which previously was available on the admin login page /admin.

Based their EVE in game roles and Pathfinders *.ini configuration, each character gets one of these 3 roles assigned:

MEMBER - default role
MANAGER - corporation admin
ADMIN - server admin

Characters with MANAGER or ADMIN roles have access to the new admin/settings page.  They can change the required role, a char must have, for some new map rights:

update map
delete map
import map
export map

Characters that don´t have the required role for a map action (e.g. "delete map") get see some warnings on their UI


Bot section edited

I removed web-bots section and edited the PTB document's link because it was invalid, and capitalized the titles

Great, Thank you so much


Add support for routines

Hi @jklingsporn ,
I'm having issue with performing select query on generated Routine i.e. user-defined PL/pgSQL function (which returns JSON type) inside of transaction() method in my project which is using  vertx-jooq-classic-reactive 5.1.1 implementation.
All jOOQ code-generation has been successfully executed when building Maven project, but problem is when I try to invoke my generated Routine this way (by Official jOOQ User Manual):
Future<JsonObject> ordersFuture = queryExecutor.transaction(transactionQE -> transactionQE
                .query(dsl -> dsl
                    .select(Routines.getAllOrders())
            ));

...I get these errors during compile-time in Eclipse IDE:

"Type mismatch: cannot convert from Future < Object > to Future < JsonObject > "

"Type mismatch: cannot convert from Future < QueryResult > to Future < Object > "

This is my Routines.java class code
/**
 * Convenience access to all stored procedures and functions in public
 */
@SuppressWarnings({ "all", "unchecked", "rawtypes" })
public class Routines {

    /**
     * Call public.get_all_orders
     */
    public static JSON getAllOrders(Configuration configuration) {
        GetAllOrders f = new GetAllOrders();

        f.execute(configuration);
        return f.getReturnValue();
    }

    /**
     * Get public.get_all_orders as a field.
     */
    public static Field<JSON> getAllOrders() {
        GetAllOrders f = new GetAllOrders();

        return f.asField();
    }

// other methods left out for code brevity
}

...and this is my GetAllOrders.java (routine) class:
/**
 * This class is generated by jOOQ.
 */
@SuppressWarnings({ "all", "unchecked", "rawtypes" })
public class GetAllOrders extends AbstractRoutine<JSON> {

    private static final long serialVersionUID = 917599810;

    /**
     * The parameter public.get_all_orders.RETURN_VALUE.
     */
    public static final Parameter<JSON> RETURN_VALUE = Internal.createParameter("RETURN_VALUE", org.jooq.impl.SQLDataType.JSON, false, false);

    /**
     * Create a new routine call instance
     */
    public GetAllOrders() {
        super("get_all_orders", Public.PUBLIC, org.jooq.impl.SQLDataType.JSON);

        setReturnParameter(RETURN_VALUE);
    }
}

Here is my plain PL/pgSQL function:
CREATE OR REPLACE FUNCTION get_all_orders() RETURNS JSON AS
$BODY$
DECLARE
    single_order RECORD;
    single_order_json json;
    orders_array json[];
BEGIN

    FOR single_order IN SELECT * FROM public.orders ORDER BY order_id
    LOOP
        SELECT get_order_by_order_id(single_order.order_id) INTO single_order_json;
        orders_array = array_append(orders_array, single_order_json);
    END LOOP;

    return (select json_build_object(
        'orders', orders_array
    ));
END;
$BODY$
LANGUAGE 'plpgsql';

I've already posted question on Stackoverflow and got answer from Lukas Eder that PROBABLY Future<QueryResult> needs to be explicitly mapped into to Future<JsonObject>.
Is there any appropriate way to handle this conversion manually, i.e. are there any existing examples to follow in order to achieve this conversion?
Thank you in advance.
P.S. I've researched for similar issues but closest one I've managed to find was this one, but it's using AsyncSQLClient while  vertx-jooq-classic-reactive 5.1.1 is using newer (Reactive) one.

I've managed to (ALMOST) fix this issue by creating <forcedType>s in pom.xml file for user-defined function RETURN_VALUE  and INPUT parameter type as described in here in Stackoverflow Q&A. Thing is that now I'm having trouble with properly extracting/mapping values from QueryResult variable and I presume (if I'm not wrong) that it has to do something with this statement mentioned in known issues:

Although postgres and the reactive driver permit false or true as valid JSON, this is not supported. JSON- and JSONB-fields are automatically mapped to a io.vertx.core.json.JsonObject (or array) which cannot handle those types.

@jklingsporn does that ALSO include generated routines fields?

Hello @nixos89,
sorry for being unresponsive, but I was busy with my daily job and this project fell a bit behind.
Unfortunately I have never used the routines-feature of jooq. You are writing:

Problem is that now I'm having is with properly extracting/mapping values from QueryResult variable
What exactly is the problem? If you can't use one of the get-methods you can call QueryResult.<Row>unwrap() to get the underlying vertx-Row and extract your object from there?

Hi @jklingsporn ,
no problem and thanks for replying. :) The thing is that I can NOT extract result of selected routine from QueryResult (above in my 1st post I've made a mistake to put it into JsonObject). I've edited my pom.xml (added forcedTypes like in above mentioned link), regenerated my jOOQ code and modifed  jOOQ-fied SELECT-clause and now it looks like this (method returns OrderService instance because I'm using service-proxies in my project now):
         @Override
	public OrderService getAllOrdersJooqSP(Handler<AsyncResult<JsonObject>> resultHandler) {
		Future<QueryResult> ordersFuture = queryExecutor.transaction(qe -> qe
			.query(dsl -> dsl.select(Routines.getAllOrders()) ));
	    ordersFuture.onComplete(handler -> {
			if (handler.succeeded()) {
				QueryResult qRes = handler.result();
				JsonObject ordersJsonObject = OrderUtilHelper.convertGetAllOrdersQRToJsonObject(qRes);
				resultHandler.handle(Future.succeededFuture(ordersJsonObject));
	    	} else {
	    		LOGGER.error("Error, something failed in retrivening ALL orders! handler.cause() = " + handler.cause());
	    		queryExecutor.rollback();
	    		resultHandler.handle(Future.failedFuture(handler.cause()));
	    	}
	    });
		return this;
	}

I've tried to use as you've suggested QueryResult.<Row>unwrap() method, but no success - it looks like this:
public static JsonObject convertGetAllOrdersQRToJsonObject(QueryResult qr) {
		JsonObject finalRes2 = new JsonObject();
		LOGGER.info("qr.hasResults() = " + qr.hasResults());
		LOGGER.info("qr.get(\"orders\", JSON.class) = " + qr.get("orders", JSON.class)); // returns "null"
		Row ordersRow = qr.<Row>unwrap(); // I'm using it HERE!!!
		String strAllOrders = ordersRow.get(String.class, 0); // returns "null"
		LOGGER.info("strAllOrders = " + strAllOrders); // returns "null" and after this LINE it HANGS and TIMEOUTs (Error: Time out after waiting 30000ms ) !!!
		JSON jooqJSON = ordersRow.get(JSON.class, 0);
		LOGGER.info("jooqJSON = " + jooqJSON);
		LOGGER.info("jooqJSON.toString() = " + jooqJSON.toString());
		JsonArray newConverterJA = new JooqJsonToVertxJsonArrayConverter().from(qr.get("orders", JSON.class));
		for (QueryResult qRes: qr.asList()) {
			LOGGER.info("qRes.toString() = " + qRes.toString()); // returns "io.github.jklingsporn.vertx.jooq.shared.reactive.ReactiveQueryResult@6eaed780"
			Row ordersRowIn = qRes.<Row>unwrap();
			Long orderId = ordersRowIn.getLong("order_id");
			LOGGER.info("(in da for-loop) orderId = " + orderId); // returns "null"
			JsonArray val1 = qRes.get("orders", JsonArray.class); // also HANGS here and returns Timed out ERROR
			finalRes2.put("orders", val1);
		}
//		JsonObject newConverterJO = new PostgresJSONVertxJsonObjectBinding().from(qr.get("orders", JSON.class));
		LOGGER.info("newConverterJA.encodePrettily() = \n" + newConverterJA.encodePrettily());
		LOGGER.info("finalRes2.encodePrettily() = \n" + finalRes2.encodePrettily());
		LOGGER.info("Moving on...");
		return new JsonObject().put("orders", newConverterJA);
	}

Following classes are other jOOQ GENERATED classes used in jOOQ-SELECT-statement:

This is com.ns.vertx.pg.jooq.routines.GetAllOrders.java class:

// This class is generated by jOOQ.
@SuppressWarnings({ "all", "unchecked", "rawtypes" })
public class GetAllOrders extends AbstractRoutine<String> {
    private static final long serialVersionUID =<PHONE_NUMBER>;

     // The parameter public.get_all_orders.RETURN_VALUE.
    public static final Parameter<String> RETURN_VALUE = Internal.createParameter("RETURN_VALUE", org.jooq.impl.SQLDataType.JSON, false, false,
org.jooq.Converter.ofNullable(org.jooq.JSON.class, String.class, Object::toString, org.jooq.JSON::valueOf));

     // Create a new routine call instance
    public GetAllOrders() {
        super("get_all_orders", Public.PUBLIC, org.jooq.impl.SQLDataType.JSON,
org.jooq.Converter.ofNullable(org.jooq.JSON.class, String.class, Object::toString, org.jooq.JSON::valueOf));
        setReturnParameter(RETURN_VALUE);
    }
}

com.ns.vertx.pg.jooq.Routines.java class:

// Convenience access to all stored procedures and functions in public
@SuppressWarnings({ "all", "unchecked", "rawtypes" })
public class Routines {

    // Call public.get_all_orders
    public static String getAllOrders(Configuration configuration) {
        GetAllOrders f = new GetAllOrders();
        f.execute(configuration);
        return f.getReturnValue();
    }

    // Get public.get_all_orders as a field.
    public static Field<String> getAllOrders() {
        GetAllOrders f = new GetAllOrders();
        return f.asField();
    }

This is link to my updated project repo.
Is it possible I'm not doing something right...or I'm missing something?
BTW, does this known issue:

Although postgres and the reactive driver permit false or true as valid JSON, this is not supported. JSON- and JSONB-fields are automatically mapped to a io.vertx.core.json.JsonObject (or array) which cannot handle those types.

... have to something with properly reading boolean types in my JSON result (of my PL/pgSQL user-defined function ~ generated routine)? Thank you in advance.

Now that I am looking into the generated Routines-class, I can see the issue.
The generated routine tries to fetch all the data using jdbc:
public static String getAllOrders(Configuration configuration) {
        GetAllOrders f = new GetAllOrders();
        **f.execute(configuration);**
        return f.getReturnValue();
    }

That is, because the generation of routines is not intercepted by vertx-jooq. As a workaround, you should be able to create a query to call the function by using queryExecutor.query(DSL.resultQuery("SELECT get_all_orders()"))

I see.. anyway, I've tried out what you suggested - replaced parameter of queryExecutor.query() method like this:
Future<QueryResult> ordersFuture = queryExecutor.transaction(qe ->
   qe.query( dsl -> dsl.resultQuery("SELECT get_all_orders()")
));

...and tried out to extract values from QueryResult instance in convertGetAllOrdersQRToJsonObject(QueryResult) method  -> no luck! Here is how that method looks like:
public static JsonObject convertGetAllOrdersQRToJsonObject(QueryResult qr) {
		LOGGER.info("qr.hasResults() = " + qr.hasResults());
		JsonArray ordersJA = qr.get("orders", JsonArray.class);
		JsonObject ordersJO = qr.get("orders", JsonObject.class);
		LOGGER.info("ordersJA = " + ordersJA);
		LOGGER.info("ordersJO = " + ordersJO);
		Row ordersRow = qr.<Row>unwrap();
		String jooqJson = ordersRow.getString("orders");
		LOGGER.info("jooqJson = " + jooqJson);

		String strAllOrders = ordersRow.get(String.class, 0);
		LOGGER.info("strAllOrders = " + strAllOrders); // NOW it returns "null" -> does NOT hang and throw Error!
		for (QueryResult qRes: qr.asList()) {
			LOGGER.info("qRes.toString() =\n" + qRes.toString());
			JsonArray ordersJA2 = qRes.get("orders", JsonArray.class);
			JsonObject ordersJO2 = qRes.get("orders", JsonObject.class);
			LOGGER.info("ordersJA2 = " + ordersJA2);
			LOGGER.info("ordersJO2 = " + ordersJO2);
			Row singleRow = qRes.<Row>unwrap();
			LOGGER.info("singleRow.getLong(\"order_id\") = " + singleRow.getLong("order_id"));
			LOGGER.info("singleRow.getColumnIndex(\"order_id\") = " + singleRow.getColumnIndex("order_id"));
			LOGGER.info("qRes.get(\"orders\", JSON.class) = " + qRes.get("orders", JSON.class));
			LOGGER.info("qRes.get(\"orders\", JSONArray.class) = " + qRes.get("orders", JSONArray.class));
			LOGGER.info("qRes.get(\"orders\", JsonArray.class) = " + qRes.get("orders", JsonArray.class));
			LOGGER.info("qRes.get(\"orders\", String.class) = " + qRes.get("orders", String.class));
			LOGGER.info("qRes.get(\"orders\", JsonObject.class) = " + qRes.get("orders", JsonObject.class));
			Row ordersRowIn = qRes.<Row>unwrap();
			Long orderId = ordersRowIn.getLong("order_id");
			LOGGER.info("(in da for-loop) orderId = " + orderId);
		}
		return new JsonObject().put("orders", newConverterJA);
	}

...and this is output I get on my terminal (when getAllOrdersJooqSP() method is executed):
15:17:28.687 [vert.x-eventloop-thread-1] INFO com.ns.vertx.pg.service.OrderServiceImpl - Passed ordersFuture...
15:17:28.983 [vert.x-eventloop-thread-1] INFO com.ns.vertx.pg.service.OrderUtilHelper - qr.hasResults() = true
15:17:28.988 [vert.x-eventloop-thread-1] INFO com.ns.vertx.pg.service.OrderUtilHelper - ordersJA = null
15:17:28.988 [vert.x-eventloop-thread-1] INFO com.ns.vertx.pg.service.OrderUtilHelper - ordersJO = null
15:17:28.988 [vert.x-eventloop-thread-1] INFO com.ns.vertx.pg.service.OrderUtilHelper - jooqJson = null
15:17:28.988 [vert.x-eventloop-thread-1] INFO com.ns.vertx.pg.service.OrderUtilHelper - strAllOrders = null
15:17:28.989 [vert.x-eventloop-thread-1] INFO com.ns.vertx.pg.service.OrderUtilHelper - qRes.toString() =
io.github.jklingsporn.vertx.jooq.shared.reactive.ReactiveQueryResult@5fee6fd
15:17:28.990 [vert.x-eventloop-thread-1] INFO com.ns.vertx.pg.service.OrderUtilHelper - ordersJA2 = null
15:17:28.990 [vert.x-eventloop-thread-1] INFO com.ns.vertx.pg.service.OrderUtilHelper - ordersJO2 = null
15:17:28.990 [vert.x-eventloop-thread-1] INFO com.ns.vertx.pg.service.OrderUtilHelper - singleRow.getLong("order_id") = null
15:17:28.990 [vert.x-eventloop-thread-1] INFO com.ns.vertx.pg.service.OrderUtilHelper - singleRow.getColumnIndex("order_id") = -1
15:17:28.990 [vert.x-eventloop-thread-1] INFO com.ns.vertx.pg.service.OrderUtilHelper - qRes.get("orders", JSON.class) = null
15:17:28.990 [vert.x-eventloop-thread-1] INFO com.ns.vertx.pg.service.OrderUtilHelper - qRes.get("orders", JSONArray.class) = null
15:17:28.990 [vert.x-eventloop-thread-1] INFO com.ns.vertx.pg.service.OrderUtilHelper - qRes.get("orders", JsonArray.class) = null
15:17:28.990 [vert.x-eventloop-thread-1] INFO com.ns.vertx.pg.service.OrderUtilHelper - qRes.get("orders", String.class) = null
15:17:28.990 [vert.x-eventloop-thread-1] INFO com.ns.vertx.pg.service.OrderUtilHelper - qRes.get("orders", JsonObject.class) = null
15:17:28.990 [vert.x-eventloop-thread-1] INFO com.ns.vertx.pg.service.OrderUtilHelper - (in da for-loop) orderId = null

BTW, please have a look at <forcedType>s from this line in project's pom.xml file for generated routines and it's types (maybe it might help).

I've tired out to print out in convertGetAllOrdersQRToJsonObject() method outside and inside of its FOR-loop Row.getColumnName(0); and got the same result which is "get_all_orders" and for each next column with index > 0 it returns null.
Is there ANY way that org.jooq.Configuration instance can be passed into  public static String getAllOrders(Configuration configuration){...} method so it can return proper value type i.e. String instead of Field<String> ?

Are you sure the routine returns results? E.g. have you tried selecting
the function on the database (using DBeaver for example)
If yes, can you try it with plain jooq?

Am Sa., 4. Juli 2020 um 10:38 Uhr schrieb Nikola Stevanović <
<EMAIL_ADDRESS>
I've tired out to print out in convertGetAllOrdersQRToJsonObject() method
outside and inside of its FOR-loop Row.getColumnName(0); and got the same
result which is "get_all_orders" and for each next column with index >
0 it returns null.
Is there ANY way that org.jooq.Configuration instance can be passed into public
static String getAllOrders(Configuration configuration){...} method so it
can return proper value type i.e. String instead of Field ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jklingsporn/vertx-jooq/issues/157#issuecomment-653739472,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABQLZXS27SFG4PO5YHILCQ3RZ3TAVANCNFSM4OGW3W2A
.

--
Jens Klingsporn
Beselerstraße 33
22607 Hamburg
Tel: 0151/70102082

Yes, I'm sure it returns result - I've tested it in pgAdmin4 and DataGrip by running SELECT get_all_orders(); query.
Tried it with plain jOOQ and it works.
Code for running it in plain main() method:

Connection connection = DriverManager.getConnection("jdbc:postgresql://localhost:5432/vertx-jooq-cr",
				"postgres", "postgres");
		DSLContext create = DSL.using(connection, SQLDialect.POSTGRES);
		Result<Record1<String>> resultR1S = create.select(Routines.getAllOrders()).fetch();
		System.out.println("resultR1S =\n" + resultR1S);

Result I get on Console (in Eclipse IDE):
resultR1S =
+--------------------------------------------------+
|get_all_orders                                    |
+--------------------------------------------------+
|{"orders" : [{"order_id" : 1, "total_price" : 2...|
+--------------------------------------------------+

As a temporary workaround I've managed to solve my use-case by using plain-jOOQ code in combination with executeBlocking(..). I did it by using try-catch-finally blocks perform jOOQ query
Connection connection = null;
		try {
			connection = DriverManager.getConnection(
					"jdbc:postgresql://localhost:5432/vertx-jooq-cr",
					"postgres", "postgres");
		} catch (SQLException e) {
			e.printStackTrace();
		} finally {
			DSLContext create = DSL.using(connection, SQLDialect.POSTGRES);
			Result<Record1<String>> resultR1S = create.select(Routines.getAllOrders()).fetch();
			String strResultFinal = resultR1S.formatJSON(
					new JSONFormat()
					.header(false)
					.recordFormat(RecordFormat.ARRAY)
			);
			final String fixedJSONString = strResultFinal
					.substring(3, strResultFinal.length() - 3)
					.replaceAll("\\\\n", "")
					.replaceAll("\\\\", "");
			JsonObject ordersJA = new JsonObject(fixedJSONString);
			resultHandler.handle(Future.succeededFuture(ordersJA));
		}

...and invoke it in inside of executeBlocking(..) method which is inside of  getAllOrdersHandler method in my HttpServerVerticle.
@jklingsporn this is only way I could get it to run and if you have any advice/suggestion to improve my (current) solution which I've describe in this comment is more than welcome. :)

Hey, I tried to run you project but it fails when executing restore.sql, e.g. COPY public.author (author_id, first_name, last_name) FROM '$$PATH$$/3984.dat'; because these files do not exist.

And one more thing: I've created a function in vertx-jooq that just performs a simple select like this:
CREATE OR REPLACE FUNCTION get_something_by_id(id INT)
   RETURNS something as $$
   declare res something;
BEGIN
   select * into res from something where "someId" = id;
  return res;
END;
$$ language 'plpgsql' STRICT;

In a simple test I can query the function like this:
dao.queryExecutor().query(dsl -> dsl.resultQuery("select get_something_by_id("+id+")"))
This will actually return a result, however adding the following code
dao.queryExecutor().query(dsl -> dsl.resultQuery("select get_something_by_id("+id+")")).map(res -> res.get(0,Something.class))
fails with the following message org.jooq.exception.DataTypeException: Cannot convert from (1,my_string,1963297861899981181,26262,2077017515,0.86091272758393556,BAZ,"{""key"":""value""}",,"[1,2,3]",,"2020-07-06 16:38:16.054") (class java.lang.String) to class generated.classic.reactive.guice.vertx.tables.pojos.Something.
Apparently the vertx-pg-client returns function results as a String.

Hey, I tried to run you project but it fails when executing restore.sql, e.g. COPY public.author (author_id, first_name, last_name) FROM '$$PATH$$/3984.dat'; because these files do not exist.

I've updated README file how to perform full restore of DB using vertx-jooq-cr-backup.tar file (my mistake, shouldn't have only put restore.sql from .tar file). Just make sure to manually restore DB before you try to rebuild project (using maven).

In a simple test I can query the function like this:
dao.queryExecutor().query(dsl -> dsl.resultQuery("select get_something_by_id("+id+")"))

Why are you using + before and after id as parameter in your code and how do you know that vertx-pg-client is the one that returns function results as String since you get  org.jooq.exception.DataTypeException execption?

No, my test was outside of yours. I just wanted to point out that - as I understand - the vertx-pg-client returns strings for routines - regardless what type you configure for them.


CUDA OOM when finetuning meta-llama/Meta-Llama-3-8B-Instruct

I was trying to finetuning Meta-Llama-3-8B-Instruct using 4 gpus with the following command:
torchrun --nproc_per_node 4 -m training.run --output_dir llama3test --model_name_or_path meta-llama/Meta-Llama-3-8B-Instruct --train_data training/toy_data --learning_rate 1e-5 --num_train_epochs 5 --per_device_train_batch_size 1 --dataloader_drop_last True --normalized True --temperature 0.02 --query_max_len 32 --passage_max_len 128 --train_group_size 2 --mode unified --attn cccc --attn_implementation sdpa --no_gen_gas --no_emb_gas --split_emb --bf16
and all 4 gpus are out of memory
W1120 13:47:19.838000 2742257 site-packages/torch/distributed/run.py:793]  W1120 13:47:19.838000 2742257 site-packages/torch/distributed/run.py:793] ***************************************** W1120 13:47:19.838000 2742257 site-packages/torch/distributed/run.py:793] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.  W1120 13:47:19.838000 2742257 site-packages/torch/distributed/run.py:793] ***************************************** 11/20/2024 13:47:23 - WARNING - __main__ -   Process rank: 0, device: cuda:0, n_gpu: 1, distributed training: True, 16-bits training: False 11/20/2024 13:47:23 - INFO - __main__ -   Training/evaluation parameters CustomTrainingArguments( _n_gpu=1, adafactor=False, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, auto_find_batch_size=False, bf16=True, bf16_full_eval=False, data_seed=None, dataloader_drop_last=True, dataloader_num_workers=0, dataloader_persistent_workers=False, dataloader_pin_memory=True, ddp_backend=None, ddp_broadcast_buffers=None, ddp_bucket_cap_mb=None, ddp_find_unused_parameters=None, ddp_timeout=1800, debug=[], deepspeed=None, disable_tqdm=False, dispatch_batches=None, do_eval=False, do_predict=False, do_train=False, emb_p_only=False, emb_q_only=False, eval_accumulation_steps=None, eval_delay=0, eval_steps=None, evaluation_strategy=no, fp16=False, fp16_backend=auto, fp16_full_eval=False, fp16_opt_level=O1, fsdp=[], fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False}, fsdp_min_num_params=0, fsdp_transformer_layer_cls_to_wrap=None, full_determinism=False, gradient_accumulation_steps=1, gradient_checkpointing=False, gradient_checkpointing_kwargs=None, greater_is_better=None, group_by_length=False, half_precision_backend=auto, hub_always_push=False, hub_model_id=None, hub_private_repo=False, hub_strategy=every_save, hub_token=<HUB_TOKEN>, ignore_data_skip=False, include_inputs_for_metrics=False, include_num_input_tokens_seen=False, include_tokens_per_second=False, jit_mode_eval=False, label_names=None, label_smoothing_factor=0.0, learning_rate=1e-05, length_column_name=length, load_best_model_at_end=False, local_rank=0, log_level=passive, log_level_replica=warning, log_on_each_node=True, logging_dir=llama3test/runs/Nov20_13-47-23_u, logging_first_step=False, logging_nan_inf_filter=True, logging_steps=500, logging_strategy=steps, lora=False, loss_gen_factor=1.0, loss_gen_type=mixed, lr_scheduler_kwargs={}, lr_scheduler_type=linear, max_grad_norm=1.0, max_steps=-1, metric_for_best_model=None, mode=unified, mp_parameters=, neftune_noise_alpha=None, negatives_cross_device=False, no_cuda=False, no_emb_gas=True, no_gen_gas=True, num_train_epochs=5.0, optim=adamw_torch, optim_args=None, output_dir=llama3test, overwrite_output_dir=False, past_index=-1, per_device_eval_batch_size=8, per_device_generative_bs=None, per_device_train_batch_size=1, prediction_loss_only=False, push_to_hub=False, push_to_hub_model_id=None, push_to_hub_organization=None, push_to_hub_token=<PUSH_TO_HUB_TOKEN>, qlora=False, ray_scope=last, remove_unused_columns=True, report_to=['wandb'], resume_from_checkpoint=None, run_name=llama3test, save_on_each_node=False, save_only_model=False, save_safetensors=False, save_steps=500, save_strategy=steps, save_total_limit=None, seed=42, skip_memory_metrics=True, split_batches=False, split_emb=True, split_emb_full=False, temperature=0.02, tf32=None, torch_compile=False, torch_compile_backend=None, torch_compile_mode=None, torchdynamo=None, tpu_metrics_debug=False, tpu_num_cores=None, use_cpu=False, use_ipex=False, use_legacy_prediction_loop=False, use_mps_device=False, warmup_ratio=0.0, warmup_steps=0, weight_decay=0.0, ) 11/20/2024 13:47:23 - INFO - __main__ -   Model parameters ModelArguments(model_name_or_path='meta-llama/Meta-Llama-3-8B-Instruct', config_name=None, tokenizer_name=None, pooling_method='weightedmean', normalized=True, attn_implementation='sdpa', attn='cccc', projection=None) 11/20/2024 13:47:23 - INFO - __main__ -   Data parameters DataArguments(train_data='training/toy_data', train_group_size=2, query_max_len=32, passage_max_len=128, generative_max_len=None, max_example_num_per_dataset=100000000, num_samples=None, use_unique_indices=False, prefixlm=False) 11/20/2024 13:47:23 - INFO - __main__ -   Using GradCache with chunk size 1 /home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: resume_downloadis deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, useforce_download=True.   warnings.warn( 11/20/2024 13:47:24 - WARNING - __main__ -   Process rank: 2, device: cuda:2, n_gpu: 1, distributed training: True, 16-bits training: False /home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: resume_downloadis deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, useforce_download=True.   warnings.warn( 11/20/2024 13:47:24 - WARNING - __main__ -   Process rank: 3, device: cuda:3, n_gpu: 1, distributed training: True, 16-bits training: False 11/20/2024 13:47:24 - WARNING - __main__ -   Process rank: 1, device: cuda:1, n_gpu: 1, distributed training: True, 16-bits training: False /home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: resume_downloadis deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, useforce_download=True.   warnings.warn( /home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: resume_downloadis deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, useforce_download=True`.
warnings.warn(
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
11/20/2024 13:47:24 - INFO - main -   Config: LlamaConfig {
"_name_or_path": "meta-llama/Meta-Llama-3-8B-Instruct",
"architectures": [
"LlamaForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 128000,
"eos_token_id": 128009,
"hidden_act": "silu",
"hidden_size": 4096,
"id2label": {
"0": "LABEL_0"
},
"initializer_range": 0.02,
"intermediate_size": 14336,
"label2id": {
"LABEL_0": 0
},
"max_position_embeddings": 8192,
"model_type": "llama",
"num_attention_heads": 32,
"num_hidden_layers": 32,
"num_key_value_heads": 8,
"pretraining_tp": 1,
"rms_norm_eps": 1e-05,
"rope_scaling": null,
"rope_theta": 500000.0,
"tie_word_embeddings": false,
"torch_dtype": "bfloat16",
"transformers_version": "4.37.2",
"use_cache": true,
"vocab_size": 128256
}
11/20/2024 13:47:24 - INFO - main -   Set pad token to bos token: <|begin_of_text|>
11/20/2024 13:47:24 - INFO - main -   Loading dataset training/toy_data/toy_data_generative.jsonl
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
11/20/2024 13:47:25 - INFO - main -   Loading dataset training/toy_data/toy_data_embedding.jsonl
11/20/2024 13:47:26 - INFO - main -   Filtering out embedding samples with too long instructions for training/toy_data/toy_data_embedding.jsonl
11/20/2024 13:47:26 - INFO - main -   Unified mode: 10 embedding samples, 10 generative samples
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00,  5.85it/s]
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00,  6.50it/s]
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00,  6.31it/s]
Created GritLM: torch.bfloat16 dtype, weightedmean pool, unified mode, cccc attn
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00,  6.44it/s]
Created GritLM: torch.bfloat16 dtype, weightedmean pool, unified mode, cccc attn
Created GritLM: torch.bfloat16 dtype, weightedmean pool, unified mode, cccc attn
Created GritLM: torch.bfloat16 dtype, weightedmean pool, unified mode, cccc attn
11/20/2024 13:47:32 - INFO - main -   Starting training
11/20/2024 13:47:34 - INFO - training.gradcache_trainer -   ***** Running training *****
11/20/2024 13:47:34 - INFO - training.gradcache_trainer -     Num examples = 10
11/20/2024 13:47:34 - INFO - training.gradcache_trainer -     Num Epochs = 5
11/20/2024 13:47:34 - INFO - training.gradcache_trainer -     Instantaneous batch size per device = 1
11/20/2024 13:47:34 - INFO - training.gradcache_trainer -     Total train batch size (w. parallel, distributed & accumulation) = 4
11/20/2024 13:47:34 - INFO - training.gradcache_trainer -     Gradient Accumulation steps = 1
11/20/2024 13:47:34 - INFO - training.gradcache_trainer -     Total optimization steps = 10
11/20/2024 13:47:34 - INFO - training.gradcache_trainer -     Number of trainable parameters = 8,030,261,248
wandb: Using wandb-core as the SDK backend.  Please refer to https://wandb.me/wandb-core for more information.
wandb: Currently logged in as: hongjizhang183 (hongjizhang183-shanghai-jiao-tong-university). Use wandb login --relogin to force relogin
wandb: Tracking run with wandb version 0.18.7
wandb: Run data is saved locally in /home/hongjizhang/gritlm/gritlm/wandb/run-20241120_134735-vujcrvoq
wandb: Run wandb offline to turn off syncing.
wandb: Syncing run leafy-moon-48
wandb: ⭐ View project at https://wandb.ai/hongjizhang183-shanghai-jiao-tong-university/huggingface
wandb: 🚀 View run at https://wandb.ai/hongjizhang183-shanghai-jiao-tong-university/huggingface/runs/vujcrvoq
0%|                                                                                | 0/10 [00:00<?, ?it/s][rank1]:[W1120 13:47:36.928805439 reducer.cpp:1400] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
[rank2]:[W1120 13:47:36.929845239 reducer.cpp:1400] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
[rank0]:[W1120 13:47:36.936399679 reducer.cpp:1400] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
[rank3]:[W1120 13:47:36.938326946 reducer.cpp:1400] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
[rank2]: Traceback (most recent call last):
[rank2]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/runpy.py", line 196, in _run_module_as_main
[rank2]:     return _run_code(code, main_globals, None,
[rank2]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/runpy.py", line 86, in _run_code
[rank2]:     exec(code, run_globals)
[rank2]:   File "/home/hongjizhang/gritlm/gritlm/training/run.py", line 438, in
[rank2]:     main()
[rank2]:   File "/home/hongjizhang/gritlm/gritlm/training/run.py", line 419, in main
[rank2]:     trainer.train()
[rank2]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/transformers/trainer.py", line 1539, in train
[rank2]:     return inner_training_loop(
[rank2]:   File "/home/hongjizhang/gritlm/gritlm/training/gradcache_trainer.py", line 766, in _inner_training_loop
[rank2]:     self.optimizer.step()
[rank2]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/accelerate/optimizer.py", line 145, in step
[rank2]:     self.optimizer.step(closure)
[rank2]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/lr_scheduler.py", line 137, in wrapper
[rank2]:     return func.get(opt, opt.class)(*args, **kwargs)
[rank2]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/optimizer.py", line 487, in wrapper
[rank2]:     out = func(*args, **kwargs)
[rank2]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/optimizer.py", line 91, in _use_grad
[rank2]:     ret = func(self, *args, **kwargs)
[rank2]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/adamw.py", line 220, in step
[rank2]:     adamw(
[rank2]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/optimizer.py", line 154, in maybe_fallback
[rank2]:     return func(*args, **kwargs)
[rank2]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/adamw.py", line 782, in adamw
[rank2]:     func(
[rank2]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/adamw.py", line 606, in _multi_tensor_adamw
[rank2]:     exp_avg_sq_sqrt = torch._foreach_sqrt(device_exp_avg_sqs)
[rank2]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 MiB. GPU 2 has a total capacity of 79.33 GiB of which 22.00 MiB is free. Including non-PyTorch memory, this process has 79.30 GiB memory in use. Of the allocated memory 77.58 GiB is allocated by PyTorch, and 501.61 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Traceback (most recent call last):
File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/hongjizhang/gritlm/gritlm/training/run.py", line 438, in
main()
File "/home/hongjizhang/gritlm/gritlm/training/run.py", line 419, in main
trainer.train()
File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/transformers/trainer.py", line 1539, in train
return inner_training_loop(
File "/home/hongjizhang/gritlm/gritlm/training/gradcache_trainer.py", line 766, in _inner_training_loop
self.optimizer.step()
File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/accelerate/optimizer.py", line 145, in step
self.optimizer.step(closure)
File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/lr_scheduler.py", line 137, in wrapper
return func.get(opt, opt.class)(*args, **kwargs)
File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/optimizer.py", line 487, in wrapper
out = func(*args, **kwargs)
File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/optimizer.py", line 91, in _use_grad
ret = func(self, *args, **kwargs)
File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/adamw.py", line 220, in step
adamw(
File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/optimizer.py", line 154, in maybe_fallback
return func(*args, **kwargs)
File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/adamw.py", line 782, in adamw
func(
File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/adamw.py", line 606, in _multi_tensor_adamw
exp_avg_sq_sqrt = torch._foreach_sqrt(device_exp_avg_sqs)
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 112.00 MiB. GPU 0 has a total capacity of 79.33 GiB of which 12.00 MiB is free. Including non-PyTorch memory, this process has 79.31 GiB memory in use. Of the allocated memory 77.72 GiB is allocated by PyTorch, and 463.67 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/runpy.py", line 196, in _run_module_as_main
[rank0]:     return _run_code(code, main_globals, None,
[rank0]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/runpy.py", line 86, in _run_code
[rank0]:     exec(code, run_globals)
[rank0]:   File "/home/hongjizhang/gritlm/gritlm/training/run.py", line 438, in
[rank0]:     main()
[rank0]:   File "/home/hongjizhang/gritlm/gritlm/training/run.py", line 419, in main
[rank0]:     trainer.train()
[rank0]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/transformers/trainer.py", line 1539, in train
[rank0]:     return inner_training_loop(
[rank0]:   File "/home/hongjizhang/gritlm/gritlm/training/gradcache_trainer.py", line 766, in _inner_training_loop
[rank0]:     self.optimizer.step()
[rank0]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/accelerate/optimizer.py", line 145, in step
[rank0]:     self.optimizer.step(closure)
[rank0]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/lr_scheduler.py", line 137, in wrapper
[rank0]:     return func.get(opt, opt.class)(*args, **kwargs)
[rank0]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/optimizer.py", line 487, in wrapper
[rank0]:     out = func(*args, **kwargs)
[rank0]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/optimizer.py", line 91, in _use_grad
[rank0]:     ret = func(self, *args, **kwargs)
[rank0]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/adamw.py", line 220, in step
[rank0]:     adamw(
[rank0]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/optimizer.py", line 154, in maybe_fallback
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/adamw.py", line 782, in adamw
[rank0]:     func(
[rank0]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/adamw.py", line 606, in _multi_tensor_adamw
[rank0]:     exp_avg_sq_sqrt = torch._foreach_sqrt(device_exp_avg_sqs)
[rank0]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 112.00 MiB. GPU 0 has a total capacity of 79.33 GiB of which 12.00 MiB is free. Including non-PyTorch memory, this process has 79.31 GiB memory in use. Of the allocated memory 77.72 GiB is allocated by PyTorch, and 463.67 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[rank1]: Traceback (most recent call last):
[rank1]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/runpy.py", line 196, in _run_module_as_main
[rank1]:     return _run_code(code, main_globals, None,
[rank1]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/runpy.py", line 86, in _run_code
[rank1]:     exec(code, run_globals)
[rank1]:   File "/home/hongjizhang/gritlm/gritlm/training/run.py", line 438, in
[rank1]:     main()
[rank1]:   File "/home/hongjizhang/gritlm/gritlm/training/run.py", line 419, in main
[rank1]:     trainer.train()
[rank1]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/transformers/trainer.py", line 1539, in train
[rank1]:     return inner_training_loop(
[rank1]:   File "/home/hongjizhang/gritlm/gritlm/training/gradcache_trainer.py", line 766, in _inner_training_loop
[rank1]:     self.optimizer.step()
[rank1]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/accelerate/optimizer.py", line 145, in step
[rank1]:     self.optimizer.step(closure)
[rank1]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/lr_scheduler.py", line 137, in wrapper
[rank1]:     return func.get(opt, opt.class)(*args, **kwargs)
[rank1]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/optimizer.py", line 487, in wrapper
[rank1]:     out = func(*args, **kwargs)
[rank1]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/optimizer.py", line 91, in _use_grad
[rank1]:     ret = func(self, *args, **kwargs)
[rank1]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/adamw.py", line 220, in step
[rank1]:     adamw(
[rank1]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/optimizer.py", line 154, in maybe_fallback
[rank1]:     return func(*args, **kwargs)
[rank1]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/adamw.py", line 782, in adamw
[rank1]:     func(
[rank1]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/adamw.py", line 606, in _multi_tensor_adamw
[rank1]:     exp_avg_sq_sqrt = torch._foreach_sqrt(device_exp_avg_sqs)
[rank1]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 112.00 MiB. GPU 1 has a total capacity of 79.33 GiB of which 38.00 MiB is free. Including non-PyTorch memory, this process has 79.28 GiB memory in use. Of the allocated memory 77.61 GiB is allocated by PyTorch, and 453.17 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[rank3]: Traceback (most recent call last):
[rank3]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/runpy.py", line 196, in _run_module_as_main
[rank3]:     return _run_code(code, main_globals, None,
[rank3]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/runpy.py", line 86, in _run_code
[rank3]:     exec(code, run_globals)
[rank3]:   File "/home/hongjizhang/gritlm/gritlm/training/run.py", line 438, in
[rank3]:     main()
[rank3]:   File "/home/hongjizhang/gritlm/gritlm/training/run.py", line 419, in main
[rank3]:     trainer.train()
[rank3]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/transformers/trainer.py", line 1539, in train
[rank3]:     return inner_training_loop(
[rank3]:   File "/home/hongjizhang/gritlm/gritlm/training/gradcache_trainer.py", line 766, in _inner_training_loop
[rank3]:     self.optimizer.step()
[rank3]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/accelerate/optimizer.py", line 145, in step
[rank3]:     self.optimizer.step(closure)
[rank3]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/lr_scheduler.py", line 137, in wrapper
[rank3]:     return func.get(opt, opt.class)(*args, **kwargs)
[rank3]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/optimizer.py", line 487, in wrapper
[rank3]:     out = func(*args, **kwargs)
[rank3]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/optimizer.py", line 91, in _use_grad
[rank3]:     ret = func(self, *args, **kwargs)
[rank3]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/adamw.py", line 220, in step
[rank3]:     adamw(
[rank3]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/optimizer.py", line 154, in maybe_fallback
[rank3]:     return func(*args, **kwargs)
[rank3]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/adamw.py", line 782, in adamw
[rank3]:     func(
[rank3]:   File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/optim/adamw.py", line 606, in _multi_tensor_adamw
[rank3]:     exp_avg_sq_sqrt = torch._foreach_sqrt(device_exp_avg_sqs)
[rank3]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 112.00 MiB. GPU 3 has a total capacity of 79.33 GiB of which 34.00 MiB is free. Including non-PyTorch memory, this process has 79.29 GiB memory in use. Of the allocated memory 77.72 GiB is allocated by PyTorch, and 442.17 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
W1120 13:47:39.700000 2742257 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 2742332 closing signal SIGTERM
W1120 13:47:39.701000 2742257 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 2742333 closing signal SIGTERM
W1120 13:47:39.702000 2742257 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 2742334 closing signal SIGTERM
wandb: 🚀 View run leafy-moon-48 at: https://wandb.ai/hongjizhang183-shanghai-jiao-tong-university/huggingface/runs/vujcrvoq
wandb: Find logs at: wandb/run-20241120_134735-vujcrvoq/logs
E1120 13:47:40.167000 2742257 site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: 1) local_rank: 3 (pid: 2742335) of binary: /home/hongjizhang/.conda/envs/gritlm/bin/python
Traceback (most recent call last):
File "/home/hongjizhang/.conda/envs/gritlm/bin/torchrun", line 8, in
sys.exit(main())
File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 355, in wrapper
return f(*args, **kwargs)
File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/distributed/run.py", line 919, in main
run(args)
File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/distributed/run.py", line 910, in run
elastic_launch(
File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 138, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/hongjizhang/.conda/envs/gritlm/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
training.run FAILED
Failures:
<NO_OTHER_FAILURES>
Root Cause (first observed failure):
[0]:
time      : 2024-11-20_13:47:39
host      : u
rank      : 3 (local_rank: 3)
exitcode  : 1 (pid: 2742335)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================`
GPU used are NVIDIA A800-SXM4-80GB which should not be OOM when loading a 8B model in bf16 precision in principle. I don't know the reason why a Llama-3-8B model takes so much memory space.

your script is not using fsdp afaict i.e. it is trying to do data parallel which won't fit; i recommend using accelerate launch with an fsdp config (https://github.com/ContextualAI/gritlm/blob/main/scripts/training/train_gritlm_7b.sh)


Clear button inside input

I have added some changes to clear button, was stylized to bootstrap and was moved to the inside of the input.

Coverage remained the same at 98.951% when pulling f1834a46477fa0599b1de1b7a37414db21e05df3 on ehvetsi:clear-button-inside-input into 18dcd7ad28663b3289cbe79a16543a7cb6f8a2f1 on charliekassel:master.

Hi,
Any problem with my changes?
Can you merge?

?

Sorry guys, is the clear-button thing working at all? Been trying for some time now and can't get it to appear.
<datepicker :clear-button-icon="'fa fa-times'" :clear-button="datepicker.clear_button">
Renders no clear button..


[enh]: add purging log query and property test

Description of changes
Summarize the changes made by this PR.

Improvements & Bug fixes

...

New functionality

...

Test plan
How are these changes tested?

[ ] Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Documentation Changes
Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs repository?

#2020  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @nicolasgere and the rest of your teammates on  Graphite


Separate font/zoom settings for script code editor and shader code editor

This commit adds Font settings for Shader Editor, allows zooming and setup fonts in script editor and shaders editor separately(its impossible to do previously)
You can look at this video and compare with the current behavior
https://youtu.be/NakXF48C6JM

I personally don't get why people would want different fonts for coding shader and coding scripts, so I don't think this is a good idea. It bloats the settings while bringing no additional value.
Please mind that the more we add settings, the more difficult it is to find the ones that are really interesting.

I unify fonts, and make shader font size invisible in editor, the only problem is changing code font size through editor settings will no longer change the shader font. I dont think its a big problem, cuz users change it as they like using zoom wheel, and I checked the zoom settings saved correctly..

I better close this PR, cuz its not been in high quality..

Will you propose another version solving the same problem ?

Maybe, or maybe not. Depends on my mood, but you free to recreate it yourself if you wish


Dll automatic copy to the Unity project unexpected behaviour / suggestions

We have several versions of AGX for several versions of Unity for several long-term projects.
I install the new version without uninstalling the previous (aware of the problems that could lead to leaving the registry values with the old values) because I want to keep the dlls of both versions.
When I open my project, I see a green log message:
New version of agxDotNet.dll located in: C:\Program Files\Algoryx\AGX-<IP_ADDRESS>\bin\x64. Copying it to current project
It detects that the version is different, deletes the new dll I copied by hand, and copies the one it thinks is the best, but not the last I installed. This is maybe a little obscure specially when you have a lot of log messages after opening a project, and this can be ignored easily, possibly breaking your project with no clue of what happened.
After this, I install the new version, uninstalling the previous. Open my project, and does nothing.
Then, I do it by hand: Go to that folder, remove the previous .dll and .meta, paste the new one and return the focus to the Unity window again and now, detects the dll, compiles, and suddenly it still detects the previous version again, not the newest, removes the dll I put by hand and copies the old one. This looks like a bug.
I have some suggestions:

Installing a new agx version without uninstalling the previous one should be an option and we should be warned about what would happen doing so. There's many situations where a project could use several versions in different projects.

[EDIT] Just a little problem with the installer. Seems that when you install a new version without uninstalling the old one (say v1 and v2), then you have v1 and v2. Then you install v2 again, it tells you to uninstall the previous one... v2... NOT v1. So v1 stays there forever. It should get rid of all older versions or leave them but update all registry and environment vars.

Copying a dll automatically is nice if your project is small and cutting edge, but not when you have several projects stalled in different versions of AGX. This should be told as an error or warning, in a different color if you will, but not do it automatically. Instead, we could have a button on the AGX menu to update the dll version. In our case, 2017.4 has a very experimental version of .Net 4, this dll update would subsequently crash Unity without changing this option. In fact, AGXUnity and AGX version pairs should be detected, and the update menu should give us the option to select which AGX version we want, including which version of AGXUnity is compatible with each. That would be really useful and less obscure, but maybe a less complicated way would be enough.

Uninstalling an AGX version shouldn't remove the .dlls needed for the projects or anyhting usable, in my opinion, so that you can keep all versions there. But that's optional.

I agree this is more of a "Developer" feature, which is useful for me when building new versions of agxDotNet.dll several times a day.
But it still indicates that you either have the wrong AGX in PATH or PATH not set at all. We're first searching for agxDotNet.dll in the current environment and if we don't find it we check the registry for an installed version. It's important that agxDotNet.dll used in your project matches agxDotNetRuntime.dll in PATH/registry.
I don't see right now doing setup_env.bat then start Unity would have such behavior? Or are you adding AGX to (global) path from the installer?

About that last comment, yes!, I'm using the script you have in your readme page. But if you change the path to the new installation of AGX (I did by error), it will grab the dll from this new folder. Maybe that's what I did wrong, you are right...
But I still think it would be great to have a popup or some other way to know about this beforehand instead of doing it automatically.
Hopefully the rest of comments are useful!  :-)  Thanks

But answering "No" in the "Would you like to update"-popup would result in wrong dll's being used. If you try to run the scene/project, Unity probably will crash due to unresolved dll dependencies.
Or do you mean that the popup would tell you that "You probably made a mistake opening this project with the current environment - restart Unity with correct AGX"?
It's not recommended to add AGX to global path, and it's pretty much undefined having several versions installed with one of them in the global environment. Do you have to have AGX in global path?


Ensure there are no parellel edges in (boost) graph

Make sure there are no parallel (duplicate) edges in use:

use setS as the edge storage type instead of vecS
do not iterate over edges between two nodes

Technically the second item above has no effect if the first one is used, but I think it adds clarity (see add_edge method documentation in https://www.boost.org/doc/libs/1_81_0/libs/graph/doc/adjacency_list.html)

I merged without updating the commit message :-(
Apologies.


Don't swallow error details in ajax service

no issue

keep the original ember-ajax behaviour rather than returning false when we hit an ajax error - we should only be using normalizeErrorResponse to normalize our error responses 😉
ensures our application code has access to the returned status code for ajax errors

LGTM :+1: Will merge once the tests have passed


checkHeaders header name needs to be lower case

Maybe README should be updated to note that header name needs to be lower case. For example X-Custom-Header won't work and needs to be x-custom-header.
req.checkHeaders('x-custom-header').optional().isEmail();
if (req.validationErrors()) return res.status(400).end();

Or, even better: make it case insensitive!

@gustavohenke even better! Thanks :+1:

I'm glad I've helped 😁
I'll let you know as soon as a version is published with this fix.


How to Stereo Calibrate more than 2 (example, 3 or n) cameras?

Hi, I was trying 3D pose estimation using $3$ cameras: i.e., first finding 2D keypoints from each of the 03 camera frames and using cv2.triangulatePoints (to triangulate and get the 3D human pose). In such a case, how to get the camera extrinsic parameters?
Specifically, suppose for $3$ cameras, should we use $[C1, C2]$, $[C2, C3]$ pairs or should we use $[C1, C2]$, $[C1, C3]$ to find the extrinsic matrix of the cameras?

Hi.
Select one of your cameras as the primary camera. Call this camera0. Then calibrate between camera0 and camera1, and camera0 and camera2 etc. So you have the transformation between cameras like this:
camera0 <---> camera1
camera0 <---> camera2
camera0 <---> cameraN
Then you can use camera0 as your coordinate frame origin or determine coordinate transform from world origin to camera0. If you use camera0 as the origin, then you just set the rotation and translation as identity matrix and 0s vector.
If camera0 and camera2, cant see the same scene, you can calibrate like this:
camera0 <---> camera1 <---> camera2
Then you can chain the transformations to obtain: camera0 <---> camera2. But this is not recommended because error will increase if you chain too many transformations.


Add Finos logo

https://www.mecabricks.com/en/models/Gejeb8RDvKX

This needs re-colouring due to availability issues with the bricks. I've re-coloured and ensured I can order ~20 sets:
https://www.bricklink.com/v2/wanted/search.page?wantedMoreID=2155376
This uses Blue for the background and Medium Blue for the background logo colour.

Colour conversion table - just for future reference https://www.eurobricks.com/forum/index.php?/forums/topic/91641-bricklinklego-color-translation-table/

Re-coloured model here: https://www.mecabricks.com/en/models/eDvVr9b0vBR


updated grafana-operator

Added 4.4 4.5 4.6 4.7 4.8 versions.
Extracted the versions table.

more recent changes are already in


Fix CMAES._get_sigma

See issue #101
Do not cast sigma to a builtin float but keep it as a torch.Tensor. Now the return type also matches with the superclass.

@NaturalGradient what do you think?

We forgot to close this, but having discussed it, @engintoklu and I had decided that sigma should remain a float. As noted by @NaturalGradient above, in the context of CMA-ES, sigma is not a variable that can be specified for each dimension (which can be true in the context of other distribution-based algorithms). Unfortunately the similar naming across algorithms may cause confusion.
Nevertheless, we appreciate you bringing this to our attention @famura!


Stanford NER Tagger fails when specifying certain directories.

I am currently working on having the path to the NER Tagger / classifier be configurable. I encountered an interesting problem. In the case where a StanfordNLP\NERTagger is instantiated with ./lib/stanford-ner-2015-04-20/... as the root of the .jar and classifier files, the tests for our NER Tagger fails. For /usr/local/bin/stanford-ner-2015-04-20/..., the tests pass. I'm looking into the cause.
php-stanford-nlp spawns a child process. I ran the command manually in a terminal and got this:
batman@epicac2:~/workspace/search-api$ java -mx300m -cp "./lib/stanford-ner-2015-04-20/stanford-ner.jar:" edu.stanford.nlp.ie.crf.CRFClassifier -loadClassifier ./lib/stanford-ner-2015-04-20/classifiers/english.all.3class.distsim.crf.ser.gz -textFile /tmp/phpnlptagnL83bI -encoding utf8
CRFClassifier invoked on Fri Jan 29 18:52:53 MST 2016 with arguments:
   -loadClassifier ./lib/stanford-ner-2015-04-20/classifiers/english.all.3class.distsim.crf.ser.gz -textFile /tmp/phpnlptagnL83bI -encoding utf8
loadClassifier=./lib/stanford-ner-2015-04-20/classifiers/english.all.3class.distsim.crf.ser.gz
encoding=utf8
textFile=/tmp/phpnlptagnL83bI
Loading classifier from ./lib/stanford-ner-2015-04-20/classifiers/english.all.3class.distsim.crf.ser.gz ... done [5.1 sec].
sample/O request/O phrase/O
CRFClassifier tagged 3 words in 1 documents at 4.10 words per second.

batman@epicac2:~/workspace/search-api$ java -mx300m -cp "/usr/local/bin/stanford-ner-2015-04-20/stanford-ner.jar:" edu.stanford.nlp.ie.crf.CRFClassifier -loadClassifier /usr/local/bin/stanford-ner-2015-04-20/classifiers/english.all.3class.distsim.crf.ser.gz -textFile /tmp/phpnlptagcZIiTV -encoding utf8
CRFClassifier invoked on Fri Jan 29 18:53:53 MST 2016 with arguments:
   -loadClassifier /usr/local/bin/stanford-ner-2015-04-20/classifiers/english.all.3class.distsim.crf.ser.gz -textFile /tmp/phpnlptagcZIiTV -encoding utf8
loadClassifier=/usr/local/bin/stanford-ner-2015-04-20/classifiers/english.all.3class.distsim.crf.ser.gz
encoding=utf8
textFile=/tmp/phpnlptagcZIiTV
Loading classifier from /usr/local/bin/stanford-ner-2015-04-20/classifiers/english.all.3class.distsim.crf.ser.gz ... done [2.9 sec].
sample/O request/O phrase/O
CRFClassifier tagged 3 words in 1 documents at 46.15 words per second.

Ostensibly, both cases should work fine. I suspect it may have to do with the pipes used to communicate between processes.

I opened this issue as my "notes", and in case anyone else wants to chime in. But I figured it out. To make php-stanford-nlp play nice with the Stanford NER Tagger, we need to use absolute paths. Closing.


add chainlens phase4 application

Project Abstract
This is the phase 4 grant application for the Chainlens (Sirato) Substrate Explorer.
Previous grants and delivery milestones are included in the application.
Grant level

[ ] Level 1:  Up to $10,000, 2 approvals
[ ] Level 2:  Up to $30,000, 3 approvals
[x] Level 3:  Unlimited, 5 approvals (for >$100k: Web3 Foundation Council approval)

Application Checklist

[x] The application template has been copied and aptly renamed (project_name.md).
[x] I have read the application guidelines.
[x] Payment details have been provided (bank details via email or BTC, Ethereum (USDC/DAI) or Polkadot/Kusama (USDT) address in the application).
[x] The software delivered for this grant will be released under an open-source license specified in the application.
[x] The initial PR contains only one commit (squash and force-push if needed).
[x] The grant will only be announced once the first milestone has been accepted (see the announcement guidelines).
[ ] I prefer the discussion of this application to take place in a private Element/Matrix channel. My username is: @_______:matrix.org (change the homeserver if you use a different one)

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thanks for the application. But milestone one currently doesn't have any details or deliveries. Could you add them?

hey @Noc2 ! the table describes deliveries for both the milestones.

hey @Noc2 the table has been split milestones-wise now.

Thanks for the update, and sorry for the late reply. Could you also add the programming language to the deliveries and maybe link to the specifications above in the document? The milestone tables are the requirements of our contracts according to our terms and conditions. Also, could you define what you mean by "various charts, labels for chain data." And a reminder to sign our terms and conditions: #1909 (comment)

hey @Noc2 ! sorry, not sure what you mean by 'maybe link to the specifications above in the document'

You currently specify the deliveries partly as part of the application but not as part of the milestone specification. For example, the milestone says, "will support configuring storage option" and the text above actually defines three storage backends. It's currently not fully clear if you actually will deliver these storage options previously mentioned. The same is true for the microservice, etc.

hey @Noc2 ! i've added details against the delivery items. please check.

Hi @Noc2, regarding your query — the commercial product Chainlens is only serving EVM chains currently (our goal is for Chainlens Substrate to reach this point in the future).
Chainlens Substrate is a separate code base and OSS. The motivation for Chainlens Substrate is to support the need for quality open-source explorers for the Substrate ecosystem.
At the current time, I'm not sure Chainlens is well-enough established to obtain treasury funding — I would anticipate once we have customers for it that would be a viable way forward to support relay chain deployments.
Hi @dsm-w3f, we don't currently have any commercial customers for the Chainlens Substrate project. We're proposing the development of features based on previous discussions with parachains, but its not feature-rich enough for them yet.
We don't currently have impact metrics within the ecosystem, beyond that it serves as an OSS explorer for projects to use. This provides alternatives to Subscan for projects which is a closed-source paid-for product.

Hello @Noc2 @dsm-w3f is the review in progress?

That said, I will ping the rest of the team again to take another look at the proposal.

Thanks for the quick reply here. Just to double-check, where can I find the latest "Chainlens Substrate" version or basically the result of the three previous grants for 150k? Is it https://substrate.sirato.xyz/. In my opinion, if you are actually aiming to develop a commercial product for substrate, you should also be willing to invest in it.

That’s the correct url.

Hi @Noc2, we are investing in the product by running up instances for teams to work with it and provide feedback that went into this grant application.
We do face a chicken and egg problem with respect to how much we can invest in the project — without paying customers we're heavily constrained by resources to work on the project. We haven't had any external funding so need to be careful in this respect.
Beyond being self-sustaining, our goal for the project is for it to be the default explorer that anyone launching a Substrate network using open-source tooling can use. This is something that the substrate ecosystem needs and we believe that this proposal gets us closer to our goal in this respect.

hello @Noc2 any updates on the application review?

hello @Noc2 @dsm-w3f ! just wanted to check whether you guys got an opportunity to review the application yet. thank you!

hello @semuelle ! thank you for taking time to review our grant application. i've made amends to the application as you requested. please have a look and let us know if there's anything else needed. thank you!

hello @dsm-w3f ! thank you for taking time in reviewing our grant application.
In our grant application, one of the equally important functionalities we wish to contribute to the substrate ecosystem  is to showcase all verified contracts on the chain which we believe is an important insight into the smart contracts side of things. End-users definitely want to know which contracts they can trust before they can interact with.
The end-users are free to perform any number of contract verifications; this obviously turns into a storage problem since we store all the uploaded artifacts on the same backend filesystem at the moment. Therefore, it is only natural to store them elsewhere to reduce load on the local storage media and so we are providing that flexibility to pick a suitable option.
We are also trying to maximize our usage of existing substrate tools to aid in things like performing contract verification using the the new cargo-contract feature and let go of our existing workaround. We use more-or-less the same strategy that this new feature employs but we want to not only use what we think should be the standard way for contract verification but also actively contribute towards its testing, bug reporting and fixing and enhancements.
Many users would also like to see substrate data skimmed into the form of quality statistics - charts, graphs, numbers, etc. - as is the case with any financial product.
We have found that these are not trivial features to architect and develop. Moreover, all of this is completely open source with no active funding source at the moment. Our expenses go into setting up infrastructure to host this development for end-users to test and these costs are not transferred over to them in any way.

hello @semuelle ! can you please get more eyes to get our grant app reviewed :) ? thank you!

Thanks for the ping, @abhijeetbhagat. I sent a reminder to the rest of the committee.

Hi @conor10 and @abhijeetbhagat. Thanks again for your application. However, the committee has decided not to support it at this point. I believe the comments above reflect the reasons why, so I won't reiterate them here.
Thank you for the work you have put into Sirato and this application. We appreciate what you have done so far and welcome you to apply again at any time in the future. In any case, best of luck going forward!


add go 1.4 back to travis matrix

OK a mojor problem is that 1.6 intorduced the {block} in golang templates..
So i would go with 1.6+ really .. get with it..


fix-pathing-for-exe-and-ec pt2

Opening as a draft until I have the correct image from konflux, fixes the client server to handle the cosign binary with the .exe extension and also fixes the missing ec binary for windows.
Jira: https://issues.redhat.com/browse/SECURESIGN-777
Jira: https://issues.redhat.com/browse/SECURESIGN-737

@lance I've got https://github.com/securesign/cosign/pull/179 merged and passing EC, If you have the time can you update the cosign image here and merge this, thanks

Seeing some error's in konflux for ec:
Error: 4 errors occurred:
	* error validating image quay.io/redhat-user-workloads/rhtas-tenant/cli/client-server-cg@sha256:77dcccbed590daf202a303f19618690ae6ff3458df5bcc622864be46ad3847c6 of component client-server-cg: load: load documents: 3 errors occurred during loading:
/tmp/ec-work-438398759/data/22f9cd6c3/rule_data.json: merge error
/tmp/ec-work-438398759/data/3c1fa1438/rule_data.json: merge error
/tmp/ec-work-438398759/data/f49be2296/rule_data.json: merge error
	* error validating image quay.io/redhat-user-workloads/rhtas-tenant/cli/gitsign@sha256:5e4ae8d729afed06f04b00cf0958893ca6b79f3a4bed927043bf74bdd60ab152 of component gitsign: load: load documents: 3 errors occurred during loading:
/tmp/ec-work-438398759/data/22f9cd6c3/rule_data.json: merge error
/tmp/ec-work-438398759/data/3c1fa1438/rule_data.json: merge error
/tmp/ec-work-438398759/data/f49be2296/rule_data.json: merge error
	* error validating image quay.io/redhat-user-workloads/rhtas-tenant/cli/client-server-re@sha256:383232ee757ca24142420ab82a04833641da9bfd27ab23725c23a6e57fed6e9e of component client-server-re: load: load documents: 3 errors occurred during loading:
/tmp/ec-work-438398759/data/22f9cd6c3/rule_data.json: merge error
/tmp/ec-work-438398759/data/3c1fa1438/rule_data.json: merge error
/tmp/ec-work-438398759/data/f49be2296/rule_data.json: merge error
	* error validating image quay.io/redhat-user-workloads/rhtas-tenant/cli/cosign@sha256:1295e3268dce50b2c8eb79fc3273aebdd03cabc964fdeabf4a06369c7d684eff of component cosign: load: load documents: 3 errors occurred during loading:
/tmp/ec-work-438398759/data/22f9cd6c3/rule_data.json: merge error
/tmp/ec-work-438398759/data/3c1fa1438/rule_data.json: merge error
/tmp/ec-work-438398759/data/f49be2296/rule_data.json: merge error

Chatting with the konflux team about this, will merge when sorted

/retest

/retest

/lgtm