Java bindings using Java Native Access

Hi,
I hope you're well.
We recently created some java bindings to native using java native access. It was created based on your python bindings and offers a good performance. It might help others that also want to use hnswlib in java projects.
Could we add a link to this implementation in your section: Other Implementation?
Thanks,
Hussama

Hi @hussamaa,
Thank you! Sure, I can add the link.

Hi @yurymalkov,
Thank you so much for your quick feedback and response.
Sorry for the spam you might have received. I closed this one opened another PR with the changes to develop.
Once again, thank you so much and have a great evening!
Best,
Hussama

Got it.
Thank you for your work and a great evening for you too!

@yurymalkov, thank you for the great job. Keep it up!


openmpi : libtool .la files are not stripped and cause dependency issue

Issue description
openmpi nix package do not strip the *.la files coming from libtool.
This generate dependency issues for other project using libtool and nix, due to libtool trying to link against the dependencies of openmpi itself ( e.g libibverbs ) when it should not.
This issue has been encountered here #17937
It is a general good practice to strip libtool *.la file in package.
Steps to reproduce
Technical details

System: (NixOS: nixos-version, Ubuntu/Fedora: lsb_release -a, ...)
Nix version: (run nix-env --version)
Nixpkgs version: (run nix-instantiate --eval '<nixpkgs>' -A lib.nixpkgsVersion)

closed with #18148


Update to rust master

One important update this picks up is toml-rs v0.1.13 which includes some recent changes to the TOML spec (shouldn't be breaking for Cargo though)

r? @huonw
(rust_highfive has picked a reviewer for you, use r? to override)

@bors: r+ b65197b

:hourglass: Testing commit b65197b with merge 73b904c...

:sunny: Test successful - cargo-linux-32, cargo-linux-64, cargo-mac-32, cargo-mac-64, cargo-win-32, cargo-win-64


How to rollback transaction when using compose

Is it possible to rollback if I use compose and outcome is invalid?
def execute
  Foo.transaction do
    compose(A)
    compose(B)
    raise ActiveRecord::Rollback if invalid?
  end
end

Currently compose would throw us out of the transaction when outcome is invalid, thus not rolling  back.
My work around is to have a variation of compose, so execution continues.
    def compose_without_throw(other, *args)
      outcome = other.run(*args)

      unless outcome.valid?
        errors.merge!(outcome.errors)
      end

      outcome.result
    end

Oh and thanks for the hard work! I met you at lunch in RubyKaigi last month :)

Yup, I remember you. 😃
I switched the code a while back from raise to throw because it's faster and it seemed more appropriate because I was thinking of this as an internal trigger inside the gem. I didn't account for the fact that it would not work with transactions.
It seems like the prudent thing to do is switch that back.

Yeah, sounds like we should revert #326. I didn't think of transactions at the time.

Just thow in some idea: would having both compose and compose! be a good idea? the ! would raise error.

I think we can just throw the error. I don't think this is something the user needs to concern themselves with. It should just work. I have a fix, I'm just working on writing a test for it. It should be out soon.

Thanks!


upgrade the golint ci step

What type of PR is this?

Uncomment only one  /kind <> line, hit enter to put that in a new line, and remove leading whitespace from that line:
/kind bug
/kind documentation
/kind enhancement
/kind good-first-issue
/kind feature
/kind question
/kind design
/sig ai
/sig iot
/sig network
/sig storage
/sig storage

What this PR does / why we need it:
Process golint error.
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:

Does this PR introduce a user-facing change?

other Note

It seems that currently the error is not caused by golang 1.18 any more. There're some files cannot pass the lint check. Maybe we can revise these files according to the error output.

Good Job.
/lgtm

It seems that currently the error is not caused by golang 1.18 any more. There're some files cannot pass the lint check. Maybe we can revise these files according to the error output.

Yes, all step are success.

/lgtm
/approve

/assign @rambohe-ch

/lgtm
/approve

/lgtm


Remind me

!remindme to [do something] in X [time unit]
or
!remindme in X [time unit] to [do something]
would reply with
@ Person, I'll remind you in X [time unit]
and then after X times ([time unit] converted to seconds)
@ Person, [do something]

Looking at this now, it just seems like it would be hell to do. What were we thinking lol?
You're more than welcome to give it a shot in Kusoneko/Nekobot and I'll try and port it... but I'm not gonna attempt this. Natural language recognition is a biiiiiiiitch.

I suggest the lazy way and just force the user to use simple splits and searches and require syntax like this !remindme "something bla bla" "5h3m26s", someone requested this for DougleyBot and once I get unlazy enough to write it over there I'll try to toss it in over here (no promises)


HDDS-5677. When use s3g and goofys , mkdir a directory but file is cr…

What changes were proposed in this pull request?
First mount bucket1
goofys --endpoint http://localhost:9878 bucket1 /mount/bucket1

Reproduce case:
when I mkdir /mount/bucket1/dir1, but I found /mount/bucket1/dir1 is a file in actually.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5677
How was this patch tested?
manual tests

@zhengchenyu is this with ozone.om.enable.filesystem.paths enabled
If so, I have also faced similar issue with S3AFileSystem and proposed similar fix.
That got closed out saying we donot support S3A FileSystem in FileSystem compat mode. When ozone.om.enable.filesystem.paths is enabled, we normalize the paths, that is causing this issue. For more info refer to the below PR discussion.
https://github.com/apache/ozone/pull/1418
if possible for now you can try with config disabled and give a try, and let us know if you are still seeing the issue.
Once bucket type feature comes in, a single ozone cluster can support both FS and complete Object Store buckets.
https://issues.apache.org/jira/browse/HDDS-5672

@bharatviswa504 Yeah it works , after ozone.om.enable.filesystem.paths is set to false, ozone.om.metadata.layout is set to SIMPLE, and apply https://github.com/apache/ozone/pull/2583.  Thanks, I will close it issue.
BTW, Since HDDS-5672, maybe s3v volume should be a fixed assignment for BucketLayout: ObjectStore.

@bharatviswa504 Yeah it works , after ozone.om.enable.filesystem.paths is set to false, ozone.om.metadata.layout is set to SIMPLE, and apply #2583. Thanks, I will close this issue.
BTW, Since HDDS-5672, maybe s3v volume should be a fixed assignment for BucketLayout: ObjectStore.

Bucket layout need to be set at. bucket level, not at volume level.

@bharatviswa504 Yeah it works , after ozone.om.enable.filesystem.paths is set to false, ozone.om.metadata.layout is set to SIMPLE, and apply #2583. Thanks, I will close this issue.
BTW, Since HDDS-5672, maybe s3v volume should be a fixed assignment for BucketLayout: ObjectStore.

Bucket layout need to be set at. bucket level, not at volume level.

I mean that all bucket in s3v volume should be set to ObjectStore BucketLayout.


Update index.html

Hey!! Did you've seen this site I made - https://github.com/LexkingOfficial/lexkingofficial.github.io ? https://lexkingofficial.github.io

Wow!

NP!! It's made using the "Bootstrap 5" Technology and Google Icons and Google Fonts!! It's made possible with these!! in text coding.


Add support for Symlink Text InputFormat containing Parquet or Orc files

https://docs.databricks.com/delta/presto-compatibility.html#step-2-configure-prestoor-to-read-generated-manifests
https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-inventory.html#storage-inventory-athena-query

I'm not sure if this is all that is required to make this work. Can you add an integration test, probably in presto-product-tests/src/main/java/io/prestosql/tests/hive

I'll fix it asap.

If you need help with adding a test for this, we can chat on Slack: https://prestosql.io/slack.html

Not relevant anymore


[Bug] Pre- and post-hooks use parse phase values during the execution phase

Is this a new bug in dbt-core?

[X] I believe this is a new bug in dbt-core
[X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior
I'm unsure if that's expected, but when I run pre/post-hook with a variable, dbt compiles the hook statement using variable value from the parse phase.
In the following example, the pre-hook will be SELECT 999 instead of SELECT 111:
{% if execute %}
    {% set val = 111 %}
{% else %}
    {% set val = 999 %}
{% endif %}

{{
    config(
      pre_hook="SELECT " ~ val
    )
}}

SELECT 1 AS id

Expected Behavior
To have execution-phase values when the hook runs.
Steps To Reproduce

Add the model:
hook_issue.sql

{% if execute %}
    {% set val = 111 %}
{% else %}
    {% set val = 999 %}
{% endif %}

{{ print("Model. Execute: " ~ execute ~ "; Value: " ~ val) }}

{{
    config(
      pre_hook="{{ print('Pre-hook.  Execute: " ~ execute ~ "; Value: " ~ val ~ "') }}",
    )
}}

SELECT 1 AS id

run dbt --no-partial-parse run --select models/hook_issue.sql
In the command line output you will see that the model will have 111 value during the execution, when hooks will still have 999.

Relevant log output
dbt --no-partial-parse run --select models/hook_issue.sql
01:02:52  Running with dbt=1.8.0
01:02:52  Registered adapter: postgres=1.8.0
Model. Execute: False; Value: 999
Pre-hook. Execute: False; Value: 999
Post-hook. Execute: False; Value: 999
01:02:54  Found 8 models, 1 snapshot, 2 sources, 1 metric, 533 macros, 1 semantic model
01:02:54
01:02:55  Concurrency: 1 threads (target='dev')
01:02:55
01:02:55  1 of 1 START sql view model analytics.hook_issue ............................... [RUN]
Model. Execute: True; Value: 111
Pre-hook. Execute: False; Value: 999
Post-hook. Execute: False; Value: 999
01:02:55  1 of 1 OK created sql view model analytics.hook_issue .......................... [CREATE VIEW in 0.11s]
01:02:55
01:02:55  Finished running 1 view model in 0 hours 0 minutes and 0.41 seconds (0.41s).
01:02:55
01:02:55  Completed successfully
01:02:55
01:02:55  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1

Environment
- OS: macOS 13.6.3
- Python: 3.11.9
- dbt-core: 1.8.0
- dbt-postgres: 1.8.0

Which database adapter are you using with dbt?
postgres
Additional Context
No response

Thanks for reaching out about this and providing such a nice example @Klimmy 🤩
This is expected, and you're not the first to be surprised. See https://github.com/dbt-labs/dbt-core/issues/6837#issuecomment-1413924647 and other linked issues for explanations.
Example
Try out the following to see how you can still get the behavior you are after.
models/my_model.sql
{# jinja for parsing phase #}
{% if execute %}
    {% set val = 111 %}
{% else %}
    {% set val = 999 %}
{% endif %}

{# jinja for execution phase #}
{% set jinja_for_execution = """
{% if execute %}
    {% set val = 111 %}
{% else %}
    {% set val = 999 %}
{% endif %}
""" %}

{{
    config(
      pre_hook=[
        jinja_for_execution ~ "{{ print('❌ Pre-hook X. Execute: " ~ execute ~ "; Value: " ~ val ~ "') }}",
        jinja_for_execution ~ "{{ print('✅ Pre-hook Y. Execute: ' ~ execute ~ '; Value: ' ~ val ~ '') }}",
        jinja_for_execution ~ "SELECT {{ val }} AS pre_hook_sql",
        ],
    )
}}

SELECT 1 AS id

dbt --no-partial-parse run -s my_model

Output:
(dbt_1.8) $ dbt --no-partial-parse run -s my_model

17:13:05  Running with dbt=1.8.0
17:13:05  Registered adapter: postgres=1.8.0
❌ Pre-hook X. Execute: False; Value: 999
✅ Pre-hook Y. Execute: False; Value: 999
17:13:06  Found 2 models, 413 macros
17:13:06
17:13:07  Concurrency: 5 threads (target='postgres')
17:13:07
17:13:07  1 of 1 START sql view model dbt_dbeatty.my_model ............................... [RUN]
❌ Pre-hook X. Execute: False; Value: 999
✅ Pre-hook Y. Execute: True; Value: 111
17:13:07  1 of 1 OK created sql view model dbt_dbeatty.my_model .......................... [CREATE VIEW in 0.13s]
17:13:07
17:13:07  Finished running 1 view model in 0 hours 0 minutes and 0.49 seconds (0.49s).
17:13:07
17:13:07  Completed successfully
17:13:07
17:13:07  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1

Since this is expected behavior rather than a bug, I'm going to close this issue as not planned.
But just let me know if you have any follow-up thoughts or questions.

No need to apologize -- glad to connect you with helpful information.


check keypoint validity before extracting descriptors

New Matlab functionality will allow users to specify their own keypoints. So, the descriptor extraction function cannot assume the provided keypoints will be valid. Perform some basic checks before attempting descriptor extraction. If extracting in pyramid mode, convert invalid octaves and scale levels to the nearest available.

Resolved in v1.2.


Fix azure scalars

All changed packages have been documented.

You can try these changes at https://cadlplayground.z22.web.core.windows.net/typespec-azure/prs/343/
Check the website changes at https://tspwebsitepr.z22.web.core.windows.net/typespec-azure/prs/343/


Editor now shows in-game names

Editor displays in-game names for skins, pets, and nameplates. Some tuple indexes were changed to dictionaries to make it easier to switch between the name and the actual ID in the program.
I didn't change hats or visors yet because there are a lot of them.

Looks good to me, just a note that usually PRs are for one feature/topic exclusively

Also side note 2, I'm pretty sure there's a way to do two-way dicts. That would be useful here I think


Rust Pico SDK

Hello!
I am about to implement this library with rust.
Are there any helpful suggestions and tips? I thought about making it an FFI but decided to make it pure rust.
Also the name pico-sdk-rs sounds like a fine name?

I think @thejpster is starting to look at that? Probably worth collaborating... (unless, as a non Rust programmer, I've misunderstood?)

I think @thejpster is starting to look at that? Probably worth collaborating... (unless, as a non Rust programmer, I've misunderstood?)

Hello @thejpster!

Hello @thejpster!

There is a Github organisation over at https://github.com/rp-rs and a Matrix channel #rp-rs if anyone wants to join in with Rust on the Pico / RP2040. I'd be very happy to talk to RPT folks about making something official too.

There is a Github organisation over at https://github.com/rp-rs and a Matrix channel #rp-rs if anyone wants to join in with Rust on the Pico / RP2040. I'd be very happy to talk to RPT folks about making something official too.


New version of svgo deletes the path node

After apdating svgo to the latest version we noticed that at least in one scenario we loose a path node in svg after its minimisation. With version of svgo 0.5.0 everything works correctly.
This is an example of svg where for some reason svgo removes the path node:
https://drive.google.com/file/d/0B5-9CLbdcvLNWERwV2d4S2VvNEk/view?usp=sharing
Is there any known reason why this might happen?
Thanks,
Olena

I don't understand what's the problem. The file you've provided is being optimized correctly (only removing redundant transform).

same problem with 0.5.1
<svg xmlns="http://www.w3.org/2000/svg" width="13" height="13">
  <g transform="translate(0,-1039.3622)">
    <path
       d="m 1,1050.3622 -1,-1 10,-10 1,1 z m -1,-10 1,-1 10,10 -1,1 z"
       style="fill:#f00;" />
  </g>
</svg>

optimized into
<svg xmlns="http://www.w3.org/2000/svg" width="13" height="13"/>

in 0.5.0 it was
<svg xmlns="http://www.w3.org/2000/svg" width="13" height="13">
    <path d="M1 11l-1-1L10 0l1 1zM0 1l1-1 10 10-1 1z" fill="#f00"/>
</svg>

Yeah, that's one fixed in b3ee51c7bcf20ded13d547f81de37e5cd353c4bd.

Will it fix same problem with this svg?
<svg xmlns="http://www.w3.org/2000/svg" width="11" height="15"><path d="M 4,1 2,13 9,13 7,1 4,1 Z"/></svg>
after svgo 0.5.1:
<svg xmlns="http://www.w3.org/2000/svg" width="11" height="15"/>

Yes

v0.5.2 with the fix has been just released.


SOAP to REST feature mediating the out sequence

Describe your problem(s)
The SOAP to REST conversion feature adds mediation to the in-sequence (using the payload factory) and in the out-sequence, it only includes the ContentType property in order to set the response type to JSON. However, there are cases in the WSDL definition which defines the response path of the API invocation as well.
For example, an XSD such as the following would indicate array elements in the response message.
<xs:sequence>
<xs:element maxOccurs="unbounded" minOccurs="0" name="arrayElement" nillable="true" type="tns:test"/></xs:sequence>

If the WSDL has the above attribute set as "unbounded" against a given element, the expectation is that the element should always be formatted as an array (and placed within [] square brackets in JSON), whether it has one or many instances.
This works fine without an issue if there are multiple items included in the SOAP response for the above element. But the case matters if there's only one item available for the above element because unless a workaround such as using an XSLT mediator is used, the JSON response will not include this as an array.
Describe your solution
As a solution including mediation to the out sequence can also be looked into while creating the API from the WSDL definition.

Optional Fields
Observed in API-Manager-2.6.0

Tagging https://github.com/wso2/product-apim/issues/7130

This is currently on hold and will be reconsidered if any requirements arise


(Preceded by #303) Different auth keys per notification send

This will look pretty massive until #303 is merged.
See #130
This seems like the quickest way to get this rolling. We need this functionality in order to move forward, so this may not be the most well thought out/elegant solution, but it definitely offloads the maintenance of keys/certs to another package and allows for more flexibility.

@AdamBolfik now that WNS is merged could you maybe rebase and clean this PR up so I can look at what it actually does?

I'm closing this, it's not mergeable as-is.

I'm going to take a shot this weekend at creating a mergable PR, it is a useful feature.


specify files to pack to avoid oclif warning

this attribute is now required in plugins otherwise a warning is displayed (though notably, the plugins still work fine with the warning)
See my blog post for the reason why this is now required: https://medium.com/@jdxcode/for-the-love-of-god-dont-use-npmignore-f93c08909d8d
for other plugins, if you run npm pack; tar -xvzf *.tgz; rm -rf package *.tgz that will print the files it's currently packing

FYI @sigmavirus24 this plugin is using Circle 1.0 which I think is going away in August. I'll work up a PR to move it to Circle 2.0 for you as it's not super straightforward

https://github.com/heroku/heroku-connect-plugin/pull/95

I'm going to close this and re-open it to trigger a new merged build

went ahead and rebased it


KNOX-2800 - Knox token revocation should also work for impersonated tokens

What changes were proposed in this pull request?
When knox.token.user.limit.exceeded.action is set to REMOVE_OLDEST, Knox tries to revoke the oldest tokens of the given user for who the token was created. It's true for impersonated tokens too.
However, to be able to revoke a token one of the following conditions should be true:

the revoker is either configured using the knox.token.renewer.whitelist parameter
or, the revoker should remove its own token (see KNOX-2664)

The second condition was not updated though when the token-impersonation was implemented. From now on Knox considers a token an own token if either the userName metadata or the createdBy metadata matches the revoker subject (prior to this change we only considered the userName metadata).
How was this patch tested?
Added a new unit test case to prevent us from regression as well as executed manual testing as follows:

configured Knox this way:

gateway.knox.token.limit.per.user = 5 (gateway-site.xml)
knox.token.user.limit.exceeded.action = REMOVE_OLDEST (homepage.xml)
 knox.token.proxyuser.guest.users = bob (homepage.xml)
 knox.token.proxyuser.guest.hosts = * (homepage.xml)

logged into the Knox Home page as guest
generated 6 tokens for user bob on the Token Generation UI
confirmed that only 5 tokens were created for bob

2022-09-07 14:26:45,257 53f0bba2-a632-4f87-bbb9-466fee64c8f8 WARN  knox.gateway (IdentityAsserterHttpServletRequestWrapper.java:scrubOfExistingPrincipalParams(212)) - Possible identity spoofing attempt - impersonation parameter removed: doAs
2022-09-07 14:26:45,258 53f0bba2-a632-4f87-bbb9-466fee64c8f8 ERROR service.knoxtoken (TokenResource.java:getAuthenticationToken(767)) - Unable to get token for user bob: token limit exceeded
2022-09-07 14:26:45,259 53f0bba2-a632-4f87-bbb9-466fee64c8f8 INFO  service.knoxtoken (TokenResource.java:getAuthenticationToken(773)) - Revoking bob's oldest token d07cdeb9...72d5bca353c4 ...
2022-09-07 14:26:45,259 53f0bba2-a632-4f87-bbb9-466fee64c8f8 INFO  token.state (AliasBasedTokenStateService.java:removeTokens(437)) - Removing token state aliases
2022-09-07 14:26:45,270 53f0bba2-a632-4f87-bbb9-466fee64c8f8 INFO  token.state (AliasBasedTokenStateService.java:removeTokens(444)) - Removed token state aliases for d07cdeb9...72d5bca353c4
2022-09-07 14:26:45,270 53f0bba2-a632-4f87-bbb9-466fee64c8f8 INFO  service.knoxtoken (TokenResource.java:revoke(591)) - Knox Token service (homepage) revoked token d07cde...a353c4 (d07cdeb9...72d5bca353c4) (renewer=guest)
2022-09-07 14:26:45,273 53f0bba2-a632-4f87-bbb9-466fee64c8f8 INFO  service.knoxtoken (TokenResource.java:getAuthenticationToken(817)) - Knox Token service (homepage) issued token eyJqa3...3wSCSw (4329fcd2...7adf1dfda2c3)
2022-09-07 14:26:48,103  INFO  token.state (AliasBasedTokenStateService.java:persistTokenState(242)) - Creating token state aliases
2022-09-07 14:26:48,426  INFO  token.state (AliasBasedTokenStateService.java:persistTokenState(251)) - Created token state aliases for 4329fcd2...7adf1dfda2c3

Cc. @MrtnBalazs


Remove unused parameters and local variables

This pull request removes all instances of unused parameters and local variables throughout the codebase. The removal of these unused elements simplifies the code, improves its readability, and reduces the potential for confusion or errors caused by obsolete or redundant code.
The changes in this pull request eliminate all instances of unused variables and parameters in the code, streamlining the implementation and making it easier to understand and maintain. By reducing clutter and unused elements, the code is more efficient, more reliable, and easier to modify or extend in the future.
In this pull request, we've refactored the code to remove all unused variables and parameters. This change makes the code more straightforward and easier to follow, while also reducing the potential for bugs or errors. By cleaning up the code and removing unnecessary elements, we're making it more maintainable and more resilient.
With this pull request, we're improving the quality of the codebase by eliminating all unused variables and parameters. These changes make the code easier to read, more reliable, and more efficient. By reducing the complexity of the implementation, we're making it easier for other developers to understand and work with the code in the future.
The changes in this pull request clean up the codebase by removing all instances of unused parameters and local variables. These changes make the code more concise and easier to work with, while also reducing the potential for confusion or errors. By simplifying the implementation and removing obsolete elements, we're making the code more robust and easier to maintain over time.

Initial/Pre-Review Thoughts
Quick glance looks like the removed parameters are just in private methods so that's good. Any protected or public methods would need to keep them for API compatibility reasons.
Questions I have:

None

Potential risks:

None

What could make the full review difficult:

Nothing


[CI] Deprecate Python3.7 and invalidate cache weekly

This PR is only a suggestion. Realized while working on https://github.com/huggingface/setfit/pull/505 that SetFit is tested on Python3.7 which is deprecated since June 2023 and not supported anymore by transformers/huggingface_hub/datasets.
What this PR do:

do not test on Python3.7 anymore
~test on Python3.11 instead~. EDIT: test on Python 3.11 are failing due to "openvino==2022.3.0" being explicitly pinned so I remove support for 3.11. Might be worth checking this and adding it in a future PR.
update READMEs and adapt code to remove Python3.7 support
(unrelated) the test CI uses some cache to avoid redownloading all dependencies for each run. However it's usually good to invalidate this cache from time to time to avoid missing breaking changes from external dependencies. In this PR I suggest to invalidate the cache every week.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Looks like CI is failing with 3 different errors:

2 in tests/span/test_model_card.py::test_model_card where model card is not exactly as expected. Should be fixed by https://github.com/huggingface/setfit/pull/505/commits/3649f36d88f5ef294a1ea3b1ec935436833cb0df (in the other PR https://github.com/huggingface/setfit/pull/505)
tests/test_trainer.py::TrainerDifferentiableHeadTest::test_trainer_max_length_exceeds_max_acceptable_length => warning is not trigger and don't know why. It also fails on https://github.com/huggingface/setfit/pull/505 which makes me think it's not related to this PR.

Hello!
Apologies for the delay! I've merged these changes into main and resolved the remaining outstanding issues as well. I expect this'll all be included in a release quite soon (this week?). Thanks a bunch for making this PR!

Tom Aarsen


[PROPOSAL][SPARK] metrics for Spark3

Purpose:
In Spark3, we can't get statistics from OptimizedPlan, due to changes introduced in spark3.
spark2:
executedPlan is a DataWritingCommandExec which wraps logicalPlan (the same object as in optimizedPlan)
When we get statistics on execution, as it's the same object, we have them in optimizedPlan also
spark3:
executedPlan is a DataWritingCommandExec which CLONE logicalPlan and wraps it (not the same object)
When we get statistics on execution, in populates only into executedPlan object
Diff QueryExecution.scala:
2.4
val plan = sparkSession.sessionState.optimizer.executeAndTrack(withCachedData.clone(), tracker)
3.1
lazy val optimizedPlan: LogicalPlan = sparkSession.sessionState.optimizer.execute(withCachedData)
Proposed implementation
As I got DataWritingCommand can be only one (at root).
Cause metrics are gathered from all tasks and grouped at the job level for specific execution.
Even if we have two separate tasks for write, metrics still would be grouped and provided for the job level.
We have a discussion with @mobuchowski and figure out, that we can just get statistics from executionPlan and put it to the output dataset, cause it's only one - it should be safe just add metrics for the first output

I want to suggest a new approach for gathering job metrics.
In short, we can gather metrics per task and then collect them on JobEnd event, we would hold metrics in listener.
org.apache.spark.scheduler.SparkListener has  onTaskEnd(taskEnd: SparkListenerTaskEnd) we can use that event to collect metrics per task.
SparkListenerTaskEnd -> TaskMetrics -> (InputMetrics, OutputMetrics) (bytesRead, recordsRead, bytesWritten, recordsWritten)
To map each task to specific job, we can use SparkListenerTaskEnd.stageId()
SparkListenerJobStart.stageIds() can provide ids to map them on task metrics.
We can store Map(jobId, Set(stageIds)) (on job_start_event) and Map(stageId, Metrics) (on task_end_event) and then on job_end_event we can get metrics and sum them up.
Proc:

we can collect input and output metrics in one way
no changes in visitors

Cons:

Increase memory usage and complexity by storing additional mappings

@collado-mike @wslulciuc @wslulciuc @pawel-big-lebowski guys, what do you think about this approach?


Flax Gets Stuck at  Starting Services Full node

OS: Windows 11
Version of Flax: latest, and last two releases tested
Issue: when starting Flax it keeps getting stuck with a circle rotating saying starting services Full Node. If i try to Reload or force reload it will let me open the wallet but then everything has rotating circles and connection status says connected.
I have tried: Tried to reinstall flax via deleting flax files in Local and Roaming
Tired deleting DB and Wallet files
Please let me know what other debug info you need.

Tried to completely wipe Flax and I am still getting stuck at the same error and I have tried a few versions of flax too. Was working for a while but I can't pinpoint when it failed.


mRender + select

yadcf filter is not able to populate select options when data is formatted in mRender. In this example, change the date format filter type to select - it won't work.
https://jsfiddle.net/0ztkz9ey/51/

You need to set the column_data_type: 'rendered_html'  for that column, but for that you must use the datatables 1.10. +
See working jsfiddle


fix(card-link): disabled styles

Related Ticket(s)
#7851
Description
Clean up card-link styles, so wc and react have matching results.
testing: check if react and web component storybooks both show the same disabled results in card link
Changelog
Changed

updated styles with correct selectors

@ariellalgilmore It looks like the card headings are still not the same?
React:

Web components:

@kennylam updated! must have been seeing a cache version on my local


Add /me endpoints for JSON access

Can't get logs to see why travis failed and tests pass for me, so merging.


 It has been fixed in versions 1.4.17 onwards. 🎉
Please use the latest version of the library.

Hi, it still happens to me in 1.4.17


Add influxdb and Grafana support

Add influxdb and Grafana support

seems future of influxdb (2.x) is complex with news language (flux).
there is some problem of cpu usage
timescaledb may be a better alternative

seems future of influxdb (2.x) is complex with news language (flux).
there is some problem of cpu usage
timescaledb may be a better alternative


SignIn not working

The SignIn URL is:
https://signin.aws.amazon.com/switchrole
The extension is supposed to fill the form and generate a submit.
Unfortunately, the extension is too restrictive and the mentioned URL is not part of the permissions it asked for.

AESR do not have to support this URL.


Go Back and Forward between Reveal Modals

I would like to navigate between all the Reveal Modals on a page. How can I accomplish this Basically a Next and Back button between Reveal
Cheers

What is the other way?!

Its a great solution if you have one or two static windows but I had 138 dynamic inline reveals getting pulled in to I was looking for a more programatic way of doing it. But I figured it out another way. thanks

What's the other way ?

Probably some function which changes the href value of two buttons based on the current index / click. Like a two way counter / number input (listen on the change event and replace the href attribute as needed).


Adding a new annotation or label is supported?

Problem

This is a kind of question but opened this ticket to confirm it for sure.
When adding a new label or annotation to a deployment or template.metadata.annotation, it is not reconciled by operator.
For example, following labels or annotations are NOT reconciled.

e.g 1
kubectl label deployments.apps -n knative-serving activator newlabel=foo

e.g 2
kubectl patch deployments.apps -n knative-serving activator -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.istio.io/inject":"true"}}}}}'

So, is it alright to add new labels or annotations by users' command? My concern is that it is a bug actually and will not be available in the future.

I believe the behavior depends on how the user adds the annotation or label. The operator will apply a three-way merge using the value of the kubectl.kubernetes.io/last-applied-configuration annotation. So if a user adds the annotation/label using kubectl apply that will of course include those in kubectl.kubernetes.io/last-applied-configuration. And because they're there and not in the operator's manifest, the operator will delete them. As long as they're not present in kubectl.kubernetes.io/last-applied-configuration the operator should leave them alone.
Make sense?

The operator should behave exactly as if some other user is invoking kubectl apply on the resource you're attempting to label/annotate. That behavior is described here: https://kubernetes.io/docs/tasks/manage-kubernetes-objects/declarative-config/#merge-patch-calculation

Thank you @jcrossley3 Yes, that makes sense.
But so, in a nutshell, it is safe to add annotation by kubectl patch deployments.apps -n knative-serving activator -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.istio.io/inject":"true"}}}}}'?
What I am worried is that the annotation might be removed during upgrade or some events. If it is not 100% sure, I would like operator to support some other way which is guaranteed.

The annotation should remain as long as the deployment exists, and it should remain even when other fields are changed -- even other annotations -- because the operator should leave it alone if it's not mentioned in its internal manifest.
Of course, if it's ever deleted and re-recreated, the annotation won't survive that, obviously.
And if kubectl apply ... is used, that won't work either.
If it's something you want to persist across those possible events, then yes, I would suggest you bring that annotation under the operator's control via either a new CR field or a Transformer in the operator itself.

BTW, if this annotation is related to istio, it could be handled as a part of our new ingress API: #92

Thank you so much @jcrossley3 It is super clear now. I am closing this.


.forceignore issue, --concise parameter isn't working as expected for force:source:status command

Summary
Remote changes are not being ignored correctly by force:source:status command when using the --concise parameter. It happens when the added/metadata updated in the org doesn't exist in local source, and is supposed to be ignored as per .forceignore. Details and steps are provided for Profiles only, so that the issue can be reproduced easily without making any changes after cloning the dreamhouse-lwc repo, but it can just as easily be replicated with any metadata type (e.g. ApexClass) by making the required entries in the .forceignore file.
As per the documentation, ignored files shouldn't be returned in the output of force:source:status command when using --concise parameter. But the metadata is reported as Remote Add. Even stranger is the fact that the same file which is incorrectly reported as Remote Add by the force:source:status command is still ignored (correctly) by force:source:pull command.
This issue came into existence when the new force:source:beta commands became generally available in version 7.144.2.
This breaks our use cases which depend on the output of the status command to decide whether a pull/push needs to take place. Without the --concise parameter to work as expected we are having to use legacy versions of the source tracking and status/pull/push commands.
Steps To Reproduce:
Repository to reproduce: dreamhouse-lwc

Clone the dreamhouse-lwc repo.
Check .forceignore file to ensure that all profiles are ignored.
Without making any code changes, create a new scratch org.
sfdx force:org:create -f config/project-scratch-def.json -s -a dreamhouse-org -v DEVHUB
Push the repo source to the new scratch org.
sfdx force:source:push
Check the source status, with complete output (without --concise parameter)
sfdx force:source:status
Check the source status, with concise output
sfdx force:source:status --concise
Try retrieving from remote.
sfdx force:source:pull

Expected result

Step # 5 force:source:status - should show the complete output, i.e. Admin profile should show up as Remote Add.
Step # 6 force:source:status --concise - shouldn't show the Admin profile as Remote Add since we are using the --concise paramter and profiles are ignored using .forceignore.
Step # 7 force:source:pull - should report No results found

Actual result

Output of step # 5 is as expected.
Output of step # 6 is not correct. Even when using --concise param with the force:source:status command, the ignored profile is being reported as Remote Add.
Interestingly, output of step # 7 is also correct. force:source:pull command reports No results found, even when status command shows the profile as Remote Add.

This indicates that the .forceignore is being interpret correctly by the pull command, but not by the status command as should happen when using the --concise param.
System Information
▶ sfdx version --verbose --json
{
"cliVersion": "sfdx-cli/7.152.0",
"architecture": "darwin-x64",
"nodeVersion": "node-v14.19.1",
"pluginVersions": [
"@oclif/plugin-autocomplete 0.3.0 (core)",
"@oclif/plugin-commands 1.3.0 (core)",
"@oclif/plugin-help 3.3.1 (core)",
"@oclif/plugin-not-found 1.2.6 (core)",
"@oclif/plugin-plugins 1.10.11 (core)",
"@oclif/plugin-update 1.5.0 (core)",
"@oclif/plugin-warn-if-update-available 1.7.3 (core)",
"@oclif/plugin-which 1.0.4 (core)",
"@salesforce/sfdx-plugin-lwc-test 0.1.7 (core)",
"alias 2.0.0 (core)",
"apex 0.12.0 (core)",
"auth 2.0.2 (core)",
"community 1.1.4 (core)",
"config 1.4.6 (core)",
"custom-metadata 1.1.0 (core)",
"data 2.0.0 (core)",
"generator 2.0.1 (core)",
"info 2.0.0 (core)",
"limits 2.0.0 (core)",
"org 1.13.1 (core)",
"salesforce-alm 54.4.0 (core)",
"schema 2.1.0 (core)",
"sfdx-cli 7.152.0 (core)",
"signups 1.1.1 (core)",
"source 1.9.9 (core)",
"telemetry 2.0.0 (core)",
"templates 54.7.0 (core)",
"trust 1.1.0 (core)",
"user 2.0.0 (core)"
],
"osVersion": "Darwin 20.6.0"
}
Additional information
The issue can be replicated with any metadata type when the following 2 conditions are true -

Metadata which is being added/updated in the org is ignored using .forceignore.
Metadata is not present in your local source.

At this point force:source:status --concise command still reports the ignored metadata as Remote Add and force:source:pull command refuses to retrieve that.

We fixed this last June and didn't update the gh issue.


Error when running vagga run on RC

Hi all!
Great work so far. I'm trying to run NodeOS using vagga within vagrant (because I'm on OSX). I downloaded both vagga RCs:
x86_64-vagga & i386-vagga
and after running vagga run I get:
thread '<main>' panicked at 'assertion failed: iter.next() == Some(RootDir)', src/path_util.rs:18
It's very possible that I'm doing something wrong. Any have any idea?
Thanks!

Wow, it's the first time I've seen something similar... I haven't tested vagrant yet (and less vagga inside vagrant), but could you be able to test the QEmu images? It's more tested, and since vagrant already is using QEmu, you wouldn't get any performance improvement of using it...

So I'm getting stuck at the KVM part.
from vagrant:
vagrant@vagrant-ubuntu-trusty-64:~/node-os-2$ sudo ./run.sh
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory

from OS X:
tylerleonhardt@Tylers-MacBook-Pro-2~/D/x86_64-pc_qemu> ./run.sh

(process:39054): GLib-WARNING **: gmem.c:482: custom memory allocation vtable not supported
"kvm" accelerator not found.
No accelerator found!

I tried this with both versions:
x86_64-pc_qemu & i386-pc_qemu
Any ideas? I'd love to get this working so that I can play around with it!

KVM is Linux only. The script should have detect that's not available and
removed it from the QEmu parameters, I'll need to fix it. Until that, edit
the run.sh file and remove it yourself by hand.
El 23/11/2015 01:58, "Tyler James Leonhardt"<EMAIL_ADDRESS>escribió:

So I'm getting stuck at the KVM part.
from vagrant:
vagrant@vagrant-ubuntu-trusty-64:~/node-os-2$ sudo ./run.sh
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
from OS X:
tylerleonhardt@Tylers-MacBook-Pro-2~/D/x86_64-pc_qemu> ./run.sh
(process:39054): GLib-WARNING **: gmem.c:482: custom memory allocation vtable not supported
"kvm" accelerator not found.
No accelerator found!
I tried this with both versions:
x86_64-pc_qemu & i386-pc_qemu
Any ideas? I'd love to get this working so that I can play around with it!
—
Reply to this email directly or view it on GitHub
https://github.com/NodeOS/NodeOS/issues/184#issuecomment-158824005.

Awesome. It's working! What's the password though?

User: nodeos
Password: nodeos
:-)
What have you done to make it work?
El 23/11/2015 08:35, "Tyler James Leonhardt"<EMAIL_ADDRESS>escribió:

Awesome. It's working! What's the password though?
[image: screen shot 2015-11-23 at 2 34 44 am]
https://cloud.githubusercontent.com/assets/2644648/11331584/c84b9418-918a-11e5-99a5-258e3412173c.png
—
Reply to this email directly or view it on GitHub
https://github.com/NodeOS/NodeOS/issues/184#issuecomment-158869681.

So I'm running on OS X and this happens:
./run.sh

(process:41638): GLib-WARNING **: gmem.c:482: custom memory allocation vtable not supported
WARNING: Image format was not specified for 'usersfs' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.

Then the emulator pops up!

So... what should I do now? I was trying to follow the npkg instructions here. But npkg isn't doing anything. I wanted to try and install a random npm package like this one: https://github.com/wooorm/syllable
Seriously though. Great work :) I'm not really a low level OS guy (more of a web dev... errr full stack dev) but I'd love to help out where I can! Or learn things. I like doing that too.

npkg is deprecated, now we use standard npm :-) Yes, we need to update the
docs a lot... :-P pull-requests are welcome :-)
El 23/11/2015 09:04, "Tyler James Leonhardt"<EMAIL_ADDRESS>escribió:

So I'm running on OS X and this happens:
./run.sh
(process:41638): GLib-WARNING **: gmem.c:482: custom memory allocation vtable not supported
WARNING: Image format was not specified for 'usersfs' and probing guessed raw.
Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
Specify the 'raw' format explicitly to remove the restrictions.
Then the emulator pops up!
[image: screen shot 2015-11-23 at 2 53 24 am]
https://cloud.githubusercontent.com/assets/2644648/11331801/62a79a82-918d-11e5-9c2d-055111081b98.png
So... what should I do now? I was trying to follow the npkg instructions
here https://node-os.com/blog/get-involved/. But npkg isn't doing
anything. I wanted to try and install a random npm package like this one:
https://github.com/wooorm/syllable
Seriously though. Great work :) I'm not really a low level OS guy (more of
a web dev... errr full stack dev) but I'd love to help out where I can! Or
learn things. I like doing that too.
—
Reply to this email directly or view it on GitHub
https://github.com/NodeOS/NodeOS/issues/184#issuecomment-158873254.

Oh nice!
So I gave this a try:
npm install bin-ncurl -g
and it seems like it didn't like it very much!

Unfortunately, I don't really know how to scroll up.. haha. I hope this is informative in some way!
Thanks!

(facepalm) My fault, QEmu by default has only 128mb and seems it's too
little for npm and other memory hungry apps. Should I increase it to 256mb?
There are more people with this problem... :-/
If you want, try by changing the QEmu arguments (again! :-P).
2015-11-23 9:13 GMT+01:00 Tyler James Leonhardt<EMAIL_ADDRESS>
Oh nice!
So I gave this a try:
npm install bin-ncurl -g
and it seems like it didn't like it very much!
[image: screen shot 2015-11-23 at 3 11 41 am]
https://cloud.githubusercontent.com/assets/2644648/11332067/1bd08f08-9190-11e5-9c9d-9d5de08d508a.png
Unfortunately, I don't really know how to scroll up.. haha. I hope this is
informative in some way!
Thanks!
—
Reply to this email directly or view it on GitHub
https://github.com/NodeOS/NodeOS/issues/184#issuecomment-158874264.

--
"Si quieres viajar alrededor del mundo y ser invitado a hablar en un monton
de sitios diferentes, simplemente escribe un sistema operativo Unix."
– Linus Tordvals, creador del sistema operativo Linux

There's another problem, and it's that bash is a sh*t but it's available
everywhere :-P On the other hand, on the development files I start it with
some Node.js scripts that correctly detect if KVM is available, but that
require to have Node.js installed on the system. Do you think is a fair
trade? If so, I could be able to change them quickly and fix this
definitely...
2015-11-23 10:29 GMT+01:00<EMAIL_ADDRESS><EMAIL_ADDRESS>
(facepalm) My fault, QEmu by default has only 128mb and seems it's too
little for npm and other memory hungry apps. Should I increase it to 256mb?
There are more people with this problem... :-/
If you want, try by changing the QEmu arguments (again! :-P).
2015-11-23 9:13 GMT+01:00 Tyler James Leonhardt<EMAIL_ADDRESS>:

Oh nice!
So I gave this a try:
npm install bin-ncurl -g
and it seems like it didn't like it very much!
[image: screen shot 2015-11-23 at 3 11 41 am]
https://cloud.githubusercontent.com/assets/2644648/11332067/1bd08f08-9190-11e5-9c9d-9d5de08d508a.png
Unfortunately, I don't really know how to scroll up.. haha. I hope this
is informative in some way!
Thanks!
—
Reply to this email directly or view it on GitHub
https://github.com/NodeOS/NodeOS/issues/184#issuecomment-158874264.

--
"Si quieres viajar alrededor del mundo y ser invitado a hablar en un
monton de sitios diferentes, simplemente escribe un sistema operativo Unix."
– Linus Tordvals, creador del sistema operativo Linux

--
"Si quieres viajar alrededor del mundo y ser invitado a hablar en un monton
de sitios diferentes, simplemente escribe un sistema operativo Unix."
– Linus Tordvals, creador del sistema operativo Linux

I increased it to 256mb. I think we're one step closer!
I tried to install a couple packages with npm.

Upping it to 256mb did something so maybe increase it?
Also, would it be possible to write the script that detects if KVM is available with a more available language? I would assume someone installing NodeOS would also have Node.js... but it's one more dependency. That's my thought at least.

That's good :-) Yes, I think we'll need to increase the memory minimal limits... It's a shame, but seems it's the only way to be able to use slap and npm... :-/ Do you want to send a pull-request and get your credit for doing so? :-)
Your errors seems to be related to the network settings, more exactly to the DNS. Did you see in the boot messages something about the network configuration? There should be a text block with ip, gateway, dns...
Regarding KVM, I think too that requiring to have Node.js just to test NodeOS is a bad idea if everybody has bash, so we could stick with it, problem is that bash is a pain in the *ss :-P But probably it's the best option. The dev script is just using a regex on /proc/cpuinfo, so it could be used grep instead...

Ok, no: auto network setting are only available on the dev script :-P Add ip=dhcp on the .append line of the QEmu arguments, this will init network on boot. I'll update these changes shortly :-)

If the pull-request I've just done passes on the CI server, both the memory and network issues will be fixed :-)
And yes, npm is working ;-)

Sorry for the delay! I'm in Florida so.... Good morning!
Did you still want the pull request for the memory minimal limits? I can add the ip=dhcp line too. :)
I would work on the regex for the grep, but I don't know what I would be looking for considering KVM is linux only :P

Ah. I see you already got to it :P

In other news, I managed to install this random npm package, syllable, and run it! :smiley:

I can document my stuff and maybe that can replace what's (here)[http://node-os.com/blog/get-involved/] since it's a bit outdated?

Yeah, I needed to add the memory thing because the usage of the reverse
proxy consume a lot of memory (about 40mb more than before), so since I was
already there I fixed the other things :-) Can you confirm me that this
works now on your machine?
By the way, I'm from Spain, so now it's 16:00 PM here... ;-)
El 24/11/2015 16:00, "Tyler James Leonhardt"<EMAIL_ADDRESS>escribió:

Ah. I see you already got to it :P
—
Reply to this email directly or view it on GitHub
https://github.com/NodeOS/NodeOS/issues/184#issuecomment-159293345.

I can document my stuff and maybe that can replace what's (here)[http://node-os.com/blog/get-involved/] since it's a bit outdated?

Yes please! :-D Don't know if we should left that info for historical purposses or remove it at all and put directly references to the wiki... what do you think?

Can you confirm me that this works now on your machine?

Are there any good "build from source" instructions lying around? I can run through those. I've just been using the RC and those changes that you mentioned (ip=dhcp & 256 mb) worked for me.

Don't know if we should left that info for historical purposses or remove it at all and put directly references to the wiki... what do you think?

Since the second example mentions npkg which is deprecated as you've mentioned, those instructions probably shouldn't be linked on the home page. But, we might as well keep them around somewhere for historical purposes!
I can draft up some Markdown for the new "Getting Started" instructions :) although I think the instructions are pretty simple! I can do qemu instructions and maybe the iso instructions (through VirtualBox), but vagga wasn't working for me in vagrant. What do you think?

Are there any good "build from source" instructions lying around? I can run through those.

They are on the README.md, but they only work on Linux. It's just download the code from GitHub and exec npm install.

I can draft up some Markdown for the new "Getting Started" instructions :) although I think the instructions are pretty simple! I can do qemu instructions and maybe the iso instructions (through VirtualBox), but vagga wasn't working for me in vagrant. What do you think?

Good idea, go for it :-)

They are on the README.md, but they only work on Linux. It's just download the code from GitHub and exec npm install.

Tried this in Vagrant and it didn't work :(

Good idea, go for it :-)
On it! When will RC2 come out with those qemu script changes?

Yes, we need to update the
docs a lot...

:3

So the run.sh in the x86_64-pc_qemu build looks like this:
#!/usr/bin/env sh

QEMU=qemu-system-x86_64

grep -q -e vmx -e svm /proc/cpuinfo
if [ 0 -eq 0 ]; then
  QEMU="$QEMU -enable-kvm"
fi

$QEMU                  \
  -vga std              \
  -m 256M               \
  -redir tcp:50080::80  \
  -redir tcp:50443::443 \
  --kernel barebones    \
  --initrd initramfs    \
  -hda     usersfs      \
  -append  "root=/dev/sda ip=dhcp vga=0x344"

grep -q -e vmx -e svm /proc/cpuinfo
if [ 0 -eq 0 ]; then
QEMU="$QEMU -enable-kvm"
fi

What is this if statement doing? haha

Shouldn't it be this:

if grep -q -e vmx -e svm /proc/cpuinfo; then
QEMU="$QEMU -enable-kvm"
fi

right?

Where does this script get generated?

Here's something pretty neat. I added -nographic \ and console=ttyAMA0,115200 console=ttyS0 to the -append and it ran in my terminal instead of in a window :)

Here's something pretty neat. I added -nographic \ and console=ttyAMA0,115200 console=ttyS0 to the -append and it ran in my terminal instead of in a window :)

Oooh fancy! Nice work!

Tried this in Vagrant and it didn't work :(

What error did you get? Have you tried to generate it on a full Ubuntu virtual machine? I haven't used Vagrant, the only thing I know about it is that maybe the vagga images would run directly on top of it, but haven't tested it yet.

On it! When will RC2 come out with those qemu script changes?

We don't follow a "traditional" releases scheudle but instead a Rolling Release: if the CI server says it's ok, it gets published :-) The RC1 was released because all the basic functionality was working and we are ready for a 1.0 version, but things are moving forward in other areas too. 1.0 will be released when all the things its roadmap and all its open issues gets fixed or moved to 1.1.

What is this if statement doing?

It's supossed to check the exit status of grep, but it has a typo since the $? should have been escaped. You have them at https://github.com/NodeOS/NodeOS/tree/master/resources/runners

Here's something pretty neat. I added -nographic \ and console=ttyAMA0,115200 console=ttyS0 to the -append and it ran in my terminal instead of in a window :)

This same trick is used on the dev scripts, but it's not being used on the pre-generated images because people usually will want to use the full system instead. Do you think this should be added too?

Away from computer right now so I'll answer the error question when I get
back to it.
I can also send you a pull request for that script if you haven't fixed it
yet :)
I like the idea of having access to it directly from my terminal but vagga
and docker should do that, no? (I've never used either yet, but am looking
to get in to it!) Although, How hard is it to give run.sh a flag like  - - terminal? This flag will just add -nographic  and console=whatever to
the qemu script. Just a thought.
Also, is there a wget-like command yet? I managed to get this npm package
to work! I successfully downloaded a file with it :)
https://github.com/angleman/wgetjs
It doesn't support running from the terminal directly, but that's easy to
add. I could fork it if you think it would be useful? :)
On Wed, Nov 25, 2015, 5:01 AM Jesús Leganés Combarro <
<EMAIL_ADDRESS>wrote:

Tried this in Vagrant and it didn't work :(
What error did you get? Have you tried to generate it on a full Ubuntu
virtual machine? I haven't used Vagrant, the only thing I know about it is
that maybe the vagga images would run directly on top of it, but
haven't tested it yet.
On it! When will RC2 come out with those qemu script changes?
We don't follow a "traditional" releases scheudle but instead a Rolling
Release: if the CI server says it's ok, it gets published :-) The RC1 was
released because all the basic functionality was working and we are ready
for a 1.0 version, but things are moving forward in other areas too. 1.0
will be released when all the things its roadmap
https://github.com/NodeOS/NodeOS/issues/37 and all its open issues
https://github.com/NodeOS/NodeOS/milestones/1.0.0 gets fixed or moved
to 1.1.
What is this if statement doing?
It's supossed to check the exit status of grep, but it has a typo since
the $? should have been escaped. You have them at
https://github.com/NodeOS/NodeOS/tree/master/resources/runners
Here's something pretty neat. I added -nographic \ and
console=ttyAMA0,115200 console=ttyS0 to the -append and it ran in my
terminal instead of in a window :)
This same trick is used on the dev scripts, but it's not being used on the
pre-generated images because people usually will want to use the full
system instead. Do you think this should be added too?
—
Reply to this email directly or view it on GitHub
https://github.com/NodeOS/NodeOS/issues/184#issuecomment-159556963.

I can also send you a pull request for that script if you haven't fixed it yet :)

I have uploaded the KVM fixes, but the options to select the graphical output are all of them for you ;-)

Also, is there a wget-like command yet? I managed to get this npm package to work! I successfully downloaded a file with it :) https://github.com/angleman/wgetjs

No, but that's the magic of NodeOS! You have it ready on npm! :-D

It doesn't support running from the terminal directly, but that's easy to add. I could fork it if you think it would be useful? :)

It would be good if you add it ;-) I propose the binary has the wget name and has the same options as the UNIX one, that makes sense and will make it easier to use :-)

Here's that vagrant error:
vagrant@vagrant-ubuntu-trusty-64:~/source/NodeOS-master$ ls
bin  Contributing.md  doc  Dockerfile  node_modules  npm-debug.log  package.json  README.md  resources  scripts  vagga.yaml
vagrant@vagrant-ubuntu-trusty-64:~/source/NodeOS-master$ npm install
npm WARN package.json<EMAIL_ADDRESS>No repository field.
npm WARN package.json<EMAIL_ADDRESS>No repository field.
npm WARN package.json<EMAIL_ADDRESS>No repository field.
npm WARN package.json<EMAIL_ADDRESS>No repository field.
npm http GET https://registry.npmjs.org/download-manager
npm http GET https://registry.npmjs.org/cpio-stream
npm http GET https://registry.npmjs.org/debugfs
npm http GET https://registry.npmjs.org/fs-extra
npm http GET https://registry.npmjs.org/suppose
npm http GET https://registry.npmjs.org/tar-stream
npm http GET https://registry.npmjs.org/publish-release
npm http 304 https://registry.npmjs.org/debugfs
npm http 304 https://registry.npmjs.org/suppose
npm http 200 https://registry.npmjs.org/fs-extra
npm http 304 https://registry.npmjs.org/download-manager
npm http 200 https://registry.npmjs.org/cpio-stream
npm http 304 https://registry.npmjs.org/tar-stream
npm ERR! Error: No compatible version found: cpio-stream@'NodeOS/cpio-stream'
npm ERR! Valid install targets:
npm ERR! ["1.0.0","1.1.0"]
npm ERR!     at installTargetsError (/usr/share/npm/lib/cache.js:719:10)
npm ERR!     at /usr/share/npm/lib/cache.js:638:10
npm ERR!     at saved (/usr/share/npm/node_modules/npm-registry-client/lib/get.js:142:7)
npm ERR!     at /usr/lib/nodejs/graceful-fs/polyfills.js:133:7
npm ERR!     at Object.oncomplete (fs.js:107:15)
npm ERR! If you need help, you may report this log at:
npm ERR!     <http://github.com/isaacs/npm/issues>
npm ERR! or email it to:
npm ERR<EMAIL_ADDRESS>
npm ERR! System Linux 3.13.0-68-generic
npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install"
npm ERR! cwd /home/vagrant/source/NodeOS-master
npm ERR! node -v v0.10.25
npm ERR! npm -v 1.3.10
npm http 304 https://registry.npmjs.org/publish-release
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR!     /home/vagrant/source/NodeOS-master/npm-debug.log
npm ERR! not ok code 0
vagrant@vagrant-ubuntu-trusty-64:~/source/NodeOS-master$

but the options to select the graphical output are all of them for you ;-)

I think I can do that! :) Gonna need to brush up on my Bash ;)

It would be good if you add it ;-) I propose the binary has the wget name and has the same options as the UNIX one, that makes sense and will make it easier to use :-)

On it! I'll look of the spec of the UNIX command and mimic it as much as I can :)

Here's that vagrant error:

npm ERR! Error: No compatible version found: cpio-stream@'NodeOS/cpio-stream'
npm ERR! Valid install targets:
npm ERR! ["1.0.0","1.1.0"]

LOL! Just today my changes got merged on the 1.1.0 version :-P Here I can see two things: you are using an old version of the code (since this got fixed today) and more important, you are using an old version of Node.js/npm:

npm ERR! node -v v0.10.25
npm ERR! npm -v 1.3.10

Since RC1 changes, it's required Node.js 0.12.x and npm > 2.0.0, update them and you should be able to generate them correctly :-)
By the way, I have set the Node.js version on the package.json file but seems it's not being checked... :-/

I think I can do that! :) Gonna need to brush up on my Bash ;)

Go for it! :-D Make a pull-request and I'll help you if you need advice ;-)

On it! I'll look of the spec of the UNIX command and mimic it as much as I can :)

Great! :-D When it'll works I'll add it to the demo nodeos user :-)

Will do!
Just a heads up. I downloaded the latest build and the script says:
grep -q -e vmx -e svm /proc/cpuinfo
if [ 0 -eq 0 ]; then
  QEMU="$QEMU -enable-kvm"
fi

I can work on that --terminal flag and include this fix in that? :)

grep -q -e vmx -e svm /proc/cpuinfo if [ 0 -eq 0 ]; then QEMU="$QEMU -enable-kvm" fi

What the...?!?

I can work on that --terminal flag and include this fix in that? :)

Ok, go for it ;-)

What the...?!?

https://github.com/NodeOS/NodeOS/blob/master/resources/runners/qemu.sh#L6 I guess the ? needs to be escaped? :dizzy_face:
Here's what I came up with that handles --terminal and -t :) Now I just need to figure out what needs to be escaped... any ideas on how to know if it needs to be escaped?
#!/usr/bin/env sh

QEMU=qemu-system-x86_64

APPEND="root=/dev/sda ip=dhcp vga=0x344"
while [ ! $# -eq 0 ]
do
    case "$1" in
        --terminal | -t)
            APPEND="$APPEND console=ttyAMA0,115200 console=ttyS0"
            QEMU="$QEMU -nographic"
            ;;
    esac
    shift
done

grep -q -e vmx -e svm /proc/cpuinfo
if [ $? -eq 0 ]; then
  QEMU="$QEMU -enable-kvm"
fi

$QEMU                  \
  -vga std              \
  -m 256M               \
  -redir tcp:50080::80  \
  -redir tcp:50443::443 \
  --kernel barebones    \
  --initrd initramfs    \
  -hda     usersfs      \
  -append  "$APPEND"

Since we only need to replace the $CPU_FAMILY variable, maybe we could find
and alternative, composing the script from several files or something, so
all don't need to be escaped... what do you think?
El 25/11/2015 23:16, "Tyler James Leonhardt"<EMAIL_ADDRESS>escribió:

What the...?!?
https://github.com/NodeOS/NodeOS/blob/master/resources/runners/qemu.sh#L6
I guess the ? needs to be escaped? [image: :dizzy_face:]
Here's what I came up with that handles --terminal and -t :) Now I just
need to figure out what needs to be escaped... any ideas on how to know if
it needs to be escaped?
#!/usr/bin/env sh
QEMU=qemu-system-x86_64
APPEND="root=/dev/sda ip=dhcp vga=0x344"
while [ ! $# -eq 0 ]
do
case "$1" in
--terminal | -t)
APPEND="$APPEND console=ttyAMA0,115200 console=ttyS0"
QEMU="$QEMU -nographic"
;;
esac
shift
done
grep -q -e vmx -e svm /proc/cpuinfo
if [ $? -eq 0 ]; then
QEMU="$QEMU -enable-kvm"
fi
$QEMU
-vga std
-m 256M
-redir tcp:50080::80
-redir tcp:50443::443
--kernel barebones
--initrd initramfs
-hda     usersfs
-append  "$APPEND"
—
Reply to this email directly or view it on GitHub
https://github.com/NodeOS/NodeOS/issues/184#issuecomment-159743889.

On the topic of wget:
https://www.gnu.org/software/wget/manual/wget.html#Option-Syntax
^ this is a lot of options haha how about I'll do just:
wget <insert your url here> for the first iteration. Then add options later. Is that okay?

Obviously yes :-)
El 25/11/2015 23:30, "Tyler James Leonhardt"<EMAIL_ADDRESS>escribió:

On the topic of wget:
https://www.gnu.org/software/wget/manual/wget.html#Option-Syntax
^ this is a lot of options haha how about I'll do just:
wget  for the first iteration. Then add options
later. Is that okay?
—
Reply to this email directly or view it on GitHub
https://github.com/NodeOS/NodeOS/issues/184#issuecomment-159746113.

I do think it's a bit strange to have to escape all of those characters. There has to be a better way. Why do they need to be escaped in the first place?

https://github.com/tylerl0706/wgetjs
Very early first iteration :)
I'm gonna add -d & --destination next :)

wget("https://github.com/tylerl0706/wgetjs/archive/master.zip");

wget-ing my version of wget...

Nice put to download NodeOS source code on the example, but it would have
been better to download a pregenerated image from a marketing point of
view... ;-) Have you done a pull-request with your changes?
El 26/11/2015 07:42, "Tyler James Leonhardt"<EMAIL_ADDRESS>escribió:

wget("https://github.com/tylerl0706/wgetjs/archive/master.zip");

wget-ing my version of wget...
—
Reply to this email directly or view it on GitHub
https://github.com/NodeOS/NodeOS/issues/184#issuecomment-159822807.

I do think it's a bit strange to have to escape all of those characters. There has to be a better way. Why do they need to be escaped in the first place?

I've just splitted the runners files so now it's not needed to escape that characters :-) Happy hacking adding the graphic flags!!! :-D

Have you done a pull-request with your changes?

I did! I'm not sure if it's still maintained. I might have to publish it to npm separately.

but it would have been better to download a pregenerated image from a marketing point of
view... ;-)

But then I'd have to submit a pull request to update the url for each release! :P

but it would have been better to download a pregenerated image from a
marketing point of
view... ;-)

But then I'd have to submit a pull request to update the url for each
release! :P

Not really, I believe GitHub has a permanent link to the latest one...
Sorry, I'm on my mobile phone, I can't be able to search it :-P

Looks like you can link to the latest release but you can't link to a file in the latest release with just a url :( http://stackoverflow.com/questions/24987542/is-there-a-link-to-github-for-downloading-a-file-in-the-latest-release-of-a-repo

Seems it's time to put somewhere a webservice to do this "a la" rawgit...
:-)
El 26/11/2015 18:19, "Tyler James Leonhardt"<EMAIL_ADDRESS>escribió:

Looks like you can link to the latest release but you can't link to a file
in the latest release with just a url :(
http://stackoverflow.com/questions/24987542/is-there-a-link-to-github-for-downloading-a-file-in-the-latest-release-of-a-repo
—
Reply to this email directly or view it on GitHub
https://github.com/NodeOS/NodeOS/issues/184#issuecomment-159962831.

https://www.npmjs.com/package/node-wget
I couldn't wait any longer :P
Wanna add in?

Just letting you know, I'm still here :) I was just on holiday for a while and am swamped with work! I'll get back in soon with the updated docs and an unzip command of some sorts :)

https://www.npmjs.com/package/node-wget

I'm so happy to see this :smile:

I'm so happy to see this

:)
Are there any other commands you think are more important? I'd love to get
a git package to work but that might be tough. Not sure yet.
On Wed, Dec 2, 2015, 12:21 AM Lite McFish<EMAIL_ADDRESS>wrote:

https://www.npmjs.com/package/node-wget
I'm so happy to see this [image: :smile:]
—
Reply to this email directly or view it on GitHub
https://github.com/NodeOS/NodeOS/issues/184#issuecomment-161185001.

Are there any other commands you think are more important? I'd love to get
a git package to work but that might be tough. Not sure yet.

Git is needed so npm can br able to download modules from GitHub, other
option would be to help to include native git support on npm itself. I
think @lite20 started to develop a git cli using some bindings for libgit2,
but don't know what's the current status.
P.D.: sorry for not answer previously to this topic, I'm really busy lately
with my bachelor thesis, trying to find what v8 change made it impossible
to use anything newer than Node.js 0.11.14 and looking for a job... :-(

Git is needed so npm can br able to download modules from GitHub

Learned that the hard way :(

@lite20 started to develop a git cli using some bindings for libgit2

Does this help at all? https://github.com/nodegit/nodegit

and looking for a job... :-(

Looking at moving to the US? ;-) If so I have a healthy list of companies to apply to and some contacts at different companies! I could help you out with that :)

Does this help at all? https://github.com/nodegit/nodegit

Yes, that's the binding, I always forgot the name :-P

Looking at moving to the US? ;-) If so I have a healthy list of companies to apply to and some contacts at different companies! I could help you out with that :)

I was thinking about doing so when I finish my bachelor thesis, but in the short term I'm looking something here at Madrid, Spain. Send me a private email anyway ;-)

Git is needed so npm can br able to download modules from GitHub, other
option would be to help to include native git support on npm itself. I
think @lite20 started to develop a git cli using some bindings for libgit2,
but don't know what's the current status.

Very slow. If anyone else would like to take a shot at it, go ahead ^_^

@lite20 & @piranna I can fiddle with it and see if I get anywhere over winter break :)
@piranna : have you gotten the chance to add wget in?

Gave npm install nodegit a try.... need more memory :/
/test # npm install nodegit

WARN engine<EMAIL_ADDRESS>wanted: {"node":">= 0.12"} (current: {"node":"0.11.14","npm":"3.4.1"})
WARN engine<EMAIL_ADDRESS>wanted: {"node":">= 0.12"} (current: {"node":"0.11.14","npm":"3.4.1"})
[  555.706696] npm invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
[  555.710390] npm cpuset=/ mems_allowed=0
[  555.711029] CPU: 0 PID: 974 Comm: npm Not tainted 4.3.0 #1
[  555.711119] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
[  555.711563]  ffff88000d5a7d18 ffff88000d5a7ae8 ffffffff812bc6b7<PHONE_NUMBER>000001
[  555.711842]  ffff88000ee40000 ffff88000d5a7b78 ffffffff811266d3 ffff88000d5a7c10
[  555.712054]  ffffffff0d5a7d18 ffffffff00000000<PHONE_NUMBER>000000<PHONE_NUMBER>000003
[  555.712301] Call Trace:
[  555.712359]  [<ffffffff812bc6b7>] dump_stack+0x45/0x5e
[  555.712359]  [<ffffffff811266d3>] dump_header.isra.9+0x6e/0x1ad
[  555.712359]  [<ffffffff812c21ef>] ? ___ratelimit+0x8f/0xf0
[  555.712359]  [<ffffffff810e06d0>] oom_kill_process+0x1c0/0x370
[  555.712359]  [<ffffffff81053882>] ? has_capability_noaudit+0x12/0x20
[  555.712359]  [<ffffffff810e0c88>] out_of_memory+0x378/0x490
[  555.712359]  [<ffffffff810e5784>] __alloc_pages_nodemask+0x844/0x940
[  555.738349]  [<ffffffff8111d360>] alloc_pages_vma+0xa0/0x170
[  555.738545]  [<ffffffff81101686>] handle_mm_fault+0xda6/0x1110
[  555.738772]  [<ffffffff810789cd>] ? pick_next_task_fair+0x38d/0x410
[  555.739065]  [<ffffffff817de708>] ? __schedule+0x268/0x75b
[  555.739113]  [<ffffffff810407ce>] __do_page_fault+0x15e/0x370
[  555.739620]  [<ffffffff81040a1c>] do_page_fault+0xc/0x10
[  555.739828]  [<ffffffff817e3782>] page_fault+0x22/0x30
[  555.740189] Mem-Info:
[  555.740525] active_anon:51991 inactive_anon:1498 isolated_anon:0
[  555.740525]  active_file:4 inactive_file:10 isolated_file:0
[  555.740525]  unevictable:3359 dirty:0 writeback:0 unstable:0
[  555.740525]  slab_reclaimable:603 slab_unreclaimable:1653
[  555.740525]  mapped:2905 shmem:4321 pagetables:352 bounce:0
[  555.740525]  free:709 free_pcp:0 free_cma:0
[  555.741953] Node 0 DMA free:1000kB min:128kB low:160kB high:192kB active_anon:13692kB inactive_anon:388kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:888kB slab_reclaimable:160kB slab_unreclaimable:248kB kernel_stack:80kB pagetables:136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4 all_unreclaimable? yes
[  555.745661] lowmem_reserve[]: 0 218 218 218
[  555.745910] Node 0 DMA32 free:1836kB min:1824kB low:2280kB high:2736kB active_anon:194272kB inactive_anon:5604kB active_file:16kB inactive_file:40kB unevictable:13436kB isolated(anon):0kB isolated(file):0kB present:245632kB managed:229472kB mlocked:0kB dirty:0kB writeback:0kB mapped:11620kB shmem:16396kB slab_reclaimable:2252kB slab_unreclaimable:6364kB kernel_stack:1040kB pagetables:1272kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:204616 all_unreclaimable? yes
[  555.747732] lowmem_reserve[]: 0 0 0 0
[  555.748072] Node 0 DMA: 6*4kB (UEM) 2*8kB (EM) 2*16kB (UM) 1*32kB (M) 2*64kB (EM) 2*128kB (UE) 2*256kB (UE) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1000kB
[  555.749495] Node 0 DMA32: 67*4kB (UEM) 94*8kB (E) 33*16kB (UE) 9*32kB (UEM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1836kB
[  555.750458] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[  555.750816] 7694 total pagecache pages
[  555.750980] 0 pages in swap cache
[  555.751145] Swap cache stats: add 0, delete 0, find 0/0
[  555.751221] Free swap  = 0kB
[  555.751221] Total swap = 0kB
[  555.751813] 65406 pages RAM
[  555.752092] 0 pages HighMem/MovableOnly
[  555.752237] 4061 pages reserved
[  555.752442] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[  555.752695] [  946]     0   946   147116     4457      31      17        0             0 exclfs
[  555.753552] [  962]     0   962   148230     8058      42      23        0             0 node
[  555.753802] [  963]     0   963   148230     7989      43      23        0             0 node
[  555.754085] [  964]     0   964   145881     4793      31      17        0             0 nodeos-reverse-
[  555.754383] [  965]     0   965   146667     4131      31      16        0             0 getty
[  555.754432] [  966]     1   966   145856     4520      31      17        0             0 nsh
[  555.754432] [  974]     1   974   167903    31775     115      67        0             0 npm
[  555.755962] Out of memory: Kill process 974 (npm) score 520 or sacrifice child
[  555.757231] Killed process 974 (npm) total-vm:671612kB, anon-rss:115924kB, file-rss:11176kB

Don't install it from inside NodeOS, we don't have support for node-gyp
yet. Try to make the git client on your main OS and when it works, let's
include it on the packages.txt file.
And no, didn't have time, I'm busy lately (remember? ;-) )
El 09/12/2015 06:54, "Tyler James Leonhardt"<EMAIL_ADDRESS>escribió:

Gave npm install nodegit a try.... need more memory :/
/test # npm install nodegit
WARN engine<EMAIL_ADDRESS>wanted: {"node":">= 0.12"} (current: {"node":"0.11.14","npm":"3.4.1"})
WARN engine<EMAIL_ADDRESS>wanted: {"node":">= 0.12"} (current: {"node":"0.11.14","npm":"3.4.1"})
[  555.706696] npm invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
[  555.710390] npm cpuset=/ mems_allowed=0
[  555.711029] CPU: 0 PID: 974 Comm: npm Not tainted 4.3.0 #1
[  555.711119] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
[  555.711563]  ffff88000d5a7d18 ffff88000d5a7ae8 ffffffff812bc6b7<PHONE_NUMBER>000001
[  555.711842]  ffff88000ee40000 ffff88000d5a7b78 ffffffff811266d3 ffff88000d5a7c10
[  555.712054]  ffffffff0d5a7d18 ffffffff00000000<PHONE_NUMBER>000000<PHONE_NUMBER>000003
[  555.712301] Call Trace:
[  555.712359]  [] dump_stack+0x45/0x5e
[  555.712359]  [] dump_header.isra.9+0x6e/0x1ad
[  555.712359]  [] ? ___ratelimit+0x8f/0xf0
[  555.712359]  [] oom_kill_process+0x1c0/0x370
[  555.712359]  [] ? has_capability_noaudit+0x12/0x20
[  555.712359]  [] out_of_memory+0x378/0x490
[  555.712359]  [] __alloc_pages_nodemask+0x844/0x940
[  555.738349]  [] alloc_pages_vma+0xa0/0x170
[  555.738545]  [] handle_mm_fault+0xda6/0x1110
[  555.738772]  [] ? pick_next_task_fair+0x38d/0x410
[  555.739065]  [] ? __schedule+0x268/0x75b
[  555.739113]  [] __do_page_fault+0x15e/0x370
[  555.739620]  [] do_page_fault+0xc/0x10
[  555.739828]  [] page_fault+0x22/0x30
[  555.740189] Mem-Info:
[  555.740525] active_anon:51991 inactive_anon:1498 isolated_anon:0
[  555.740525]  active_file:4 inactive_file:10 isolated_file:0
[  555.740525]  unevictable:3359 dirty:0 writeback:0 unstable:0
[  555.740525]  slab_reclaimable:603 slab_unreclaimable:1653
[  555.740525]  mapped:2905 shmem:4321 pagetables:352 bounce:0
[  555.740525]  free:709 free_pcp:0 free_cma:0
[  555.741953] Node 0 DMA free:1000kB min:128kB low:160kB high:192kB active_anon:13692kB inactive_anon:388kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:888kB slab_reclaimable:160kB slab_unreclaimable:248kB kernel_stack:80kB pagetables:136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4 all_unreclaimable? yes
[  555.745661] lowmem_reserve[]: 0 218 218 218
[  555.745910] Node 0 DMA32 free:1836kB min:1824kB low:2280kB high:2736kB active_anon:194272kB inactive_anon:5604kB active_file:16kB inactive_file:40kB unevictable:13436kB isolated(anon):0kB isolated(file):0kB present:245632kB managed:229472kB mlocked:0kB dirty:0kB writeback:0kB mapped:11620kB shmem:16396kB slab_reclaimable:2252kB slab_unreclaimable:6364kB kernel_stack:1040kB pagetables:1272kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:204616 all_unreclaimable? yes
[  555.747732] lowmem_reserve[]: 0 0 0 0
[  555.748072] Node 0 DMA: 64kB (UEM) 28kB (EM) 216kB (UM) 132kB (M) 264kB (EM) 2128kB (UE) 2256kB (UE) 0512kB 01024kB 02048kB 04096kB = 1000kB
[  555.749495] Node 0 DMA32: 674kB (UEM) 948kB (E) 3316kB (UE) 932kB (UEM) 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 04096kB = 1836kB
[  555.750458] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[  555.750816] 7694 total pagecache pages
[  555.750980] 0 pages in swap cache
[  555.751145] Swap cache stats: add 0, delete 0, find 0/0
[  555.751221] Free swap  = 0kB
[  555.751221] Total swap = 0kB
[  555.751813] 65406 pages RAM
[  555.752092] 0 pages HighMem/MovableOnly
[  555.752237] 4061 pages reserved
[  555.752442] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[  555.752695] [  946]     0   946   147116     4457      31      17        0             0 exclfs
[  555.753552] [  962]     0   962   148230     8058      42      23        0             0 node
[  555.753802] [  963]     0   963   148230     7989      43      23        0             0 node
[  555.754085] [  964]     0   964   145881     4793      31      17        0             0 nodeos-reverse-
[  555.754383] [  965]     0   965   146667     4131      31      16        0             0 getty
[  555.754432] [  966]     1   966   145856     4520      31      17        0             0 nsh
[  555.754432] [  974]     1   974   167903    31775     115      67        0             0 npm
[  555.755962] Out of memory: Kill process 974 (npm) score 520 or sacrifice child
[  555.757231] Killed process 974 (npm) total-vm:671612kB, anon-rss:115924kB, file-rss:11176kB
—
Reply to this email directly or view it on GitHub
https://github.com/NodeOS/NodeOS/issues/184#issuecomment-163118690.

Finally got nodegit to do something on an Ubuntu VM I had. I'm not sure why it gave me so much history trouble, but we're 1 step closer.
Do you know if npm just needs the ability to clone a repo when you want to npm install something from a git repo?
If so I was thinking about just getting git clone to work first, then we'd at least have that. Then I can continue to add to the cli over time.
I thought about making a nice and simple npm package for turning a command (as a string, for example npm install - g nodegit) into an object with all the flags handled as booleans or if the flags have additional params, allow for that.

Ignore the word "history"... Really don't know where that came from. (I'm on mobile and can't edit :/)

Do you know if npm just needs the ability to clone a repo when you want to npm install something from a git repo?

You can get more info at https://github.com/NodeOS/NodeOS/issues/67

I thought about making a nice and simple npm package for turning a command (as a string, for example npm install - g nodegit) into an object with all the flags handled as booleans or if the flags have additional params, allow for that.

An arguments parser? There's a lot of them on npm. For simple use cases, minimist is just enough.

You can get more info at #67

Thanks!

An arguments parser? There's a lot of them on npm. For simple use cases, minimist is just enough.

I had a feeling. There's an npm package for everything!.... Except a node package with a cli :( .... for now!

Any idea what this might be?
x86_64-pc_qemu  ./run.sh
grep: /proc/cpuinfo: No such file or directory

(process:57817): GLib-WARNING **: gmem.c:482: custom memory allocation vtable not supported
qemu-system-x86_64: could not set up host forwarding rule 'tcp:50443::443'
qemu-system-x86_64: Device 'user' could not be initialized

I haven't really touched it in a while :/

grep: /proc/cpuinfo: No such file or directory

Now run.sh detect if the system support KVM to enable it. It should not show that error message, maybe the OSX version of grep don't support the -q flag? Could you be able to review it on the man page?

from man:
-q, --quiet, --silent
             Quiet mode: suppress normal output.  grep will only search a file until a match has been found, making searches potentially less expensive.

What's strange is that this exact file was working like a week ago when I ran it. (Or so I thought :/ )
My run.sh:
#!/usr/bin/env sh

QEMU=qemu-system-x86_64

APPEND="root=/dev/sda ip=dhcp vga=0x344"
while [ ! $# -eq 0 ]
do
    case "$1" in
        --terminal | -t)
            APPEND="$APPEND console=ttyAMA0,115200 console=ttyS0"
            QEMU="$QEMU -nographic"
            echo "$APPEND $QEMU"
            ;;
    esac
    shift
done

grep -q -e vmx -e svm /proc/cpuinfo
if [ $? -eq 0 ]; then
  QEMU="$QEMU -enable-kvm"
fi

$QEMU                  \
  -vga std              \
  -m 512M               \
  -redir tcp:50080::80  \
  -redir tcp:50443::443 \
  --kernel barebones    \
  --initrd initramfs    \
  -hda     usersfs      \
  -append  "$APPEND"

By the way, do you have your git client somewhere? And don't include it inside nodegit package and do it in an independent package, they told me they don't want to add a cli but instead left it as a library only module.

What's strange is that this exact file was working like a week ago when I ran it. (Or so I thought :/ )
My run.sh:

The same and me. As far as it works, I would not worry too much about this, we could get the same behaviour by redirecting to /dev/null.

they told me they don't want to add a cli but instead left it as a library only module.

So I would make another npm package that has a cli that uses nodegit?

So I would make another npm package that has a cli that uses nodegit?

Yes, that's it :-)

The same and me. As far as it works, I would not worry too much about this, we could get the same behaviour by redirecting to /dev/null.

Maybe I'll just try a fresh install. It's been a while.

closing because target problem was resolved.


AttributeError: module 'bitsandbytes' has no attribute 'nn'

I am working with Amazon Sagemaker with a ml.g5.12x large instance.
In my notebook I install bitsandbytes as follows:
!pip uninstall bitsandbytes -y
!pip install bitsandbytes==0.40.2 #bitsandbytes
%pip install torch
import bitsandbytes
import torch
!python -m bitsandbytes

The final line gives me
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++ BUG REPORT INFORMATION ++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++ /usr/local CUDA PATHS +++++++++++++++++++
+++++++++++++++ WORKING DIRECTORY CUDA PATHS +++++++++++++++
++++++++++++++++++ LD_LIBRARY CUDA PATHS +++++++++++++++++++
/opt/conda/lib/python3.10/site-packages/nvidia/cuda_runtime/lib/ CUDA PATHS
++++++++++++++++++++++++++ OTHER +++++++++++++++++++++++++++
COMPILED_WITH_CUDA = True
COMPUTE_CAPABILITIES_PER_GPU = ['8.6', '8.6', '8.6', '8.6']
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++ DEBUG INFO END ++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Running a quick check that:
+ library is importable
+ CUDA function is callable
WARNING: Please be sure to sanitize sensible info from any such env vars!
SUCCESS!
Installation was successful!
I then try:
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
  'decapoda-research/llama-7b-hf,
  device_map='auto',
  load_in_8bit=True,
  max_memory=f'{int(torch.cuda.mem_get_info()[0]/1024**3)-2}GB')

and get AttributeError: module 'bitsandbytes' has no attribute 'nn'
This example comes straight from the ReadMe.
I also tried the following approach:
model_id = "tiiuae/falcon-40b"
bnb_config = BitsAndBytesConfig(
	load_in_4bit=True,
	bnb_4bit_use_double_quant=True,
	bnb_4bit_quant_type="nf4",
	bnb_4bit_compute_dtype=torch.bfloat16
)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, quantization_config=bnb_config, device_map="auto")

Which results in the exact same AttributeError: module 'bitsandbytes' has no attribute 'nn'
Any suggestions on how I can get my code to work? Thanks in advance for your time.
EDIT: I realised below code might be important for this issue too.
# Add installed cuda runtime to path for bitsandbytes
cuda_install_dir = '/'.join(nvidia.__file__.split('/')[:-1]) + '/cuda_runtime/lib/'
os.environ['LD_LIBRARY_PATH'] =  cuda_install_dir

EDIT 2:
Maybe it goes wrong because of torch.nn?
import textract
import textract
import pandas as pd
import random
from random import randrange
from datetime import timedelta
from datetime import datetime
import string
import bitsandbytes
import torch
import torch.nn as nn
from transformers import BitsAndBytesConfig, AutoTokenizer, AutoConfig, AutoModelForCausalLM
import os
import nvidia

I face the same problem. I have fine-tuned the  Llama-2-7b-chat-hf following the instructions given in this notebook: https://colab.research.google.com/drive/1PEQyJO1-f6j0S_XJ8DV50NkpzasXkrzd?usp=sharing#scrollTo=ib_We3NLtj2E
After finished training I reload the Llama-2-7b-chat-hf and merge it with the LORA weights.  Then I save the model to disk.
When I reload it I use the exact specifications for 'bnb_config' as @WouterMulders1999 and get the error:
[Previous line repeated 1 more time]
File "/home/user/miniconda3/envs/env_name/lib/python3.11/site-packages/transformers/integrations/bitsandbytes.py", line 158, in _replace_with_bnb_linear
model._modules[name] = bnb.nn.Linear4bit(
^^^^^^
AttributeError: module 'bitsandbytes' has no attribute 'nn'
When I "remerge" the models using:
base_model = AutoModelForCausalLM.from_pretrained(
model_name,
low_cpu_mem_usage=True,
return_dict=True,
torch_dtype=torch.float16,
device_map=device_map,
)
model = PeftModel.from_pretrained(base_model, new_model)
model = model.merge_and_unload()
Inference works in jupyter notebook but I get the same error in VScode. When trying to load the finished model from disk in jupyter the kernel dies.
The environment was set up as:
!pip install -q -U bitsandbytes
!pip install -q -U git+https://github.com/huggingface/transformers.git
!pip install -q -U git+https://github.com/huggingface/peft.git
!pip install -q -U git+https://github.com/huggingface/accelerate.git
!pip install -q datasets
!pip install -q -U einops
!pip install -q -U safetensors
!pip install -q -U torch
!pip install -q -U xformers
!pip install -q -U scipy
!pip install -q -U trl
!pip install -q -U tensorboard
!pip install -q -U tensorboardX

@TimDettmers the same issue on Google Compute VM ubuntu 20.04. Any idea what could be wrong?
AttributeError: module 'bitsandbytes' has no attribute 'nn'
Fails this method from peft:
def is_bnb_4bit_available() -> bool:
    if not is_bnb_available():
        return False

    import bitsandbytes as bnb

    return hasattr(bnb.nn, "Linear4bit")

But when I do on the poetry venv just from bitsandbytes import nn it works.
Libs:
peft                      0.6.2
bitsandbytes              0.41.2.post2
torch                     2.1.1
torchvision               0.16.1
transformers              4.35.2

Here is the output of python -m bitsandbytes:
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++ BUG REPORT INFORMATION ++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

++++++++++++++++++ /usr/local CUDA PATHS +++++++++++++++++++
/usr/local/cuda-12.3/targets/x86_64-linux/lib/stubs/libcuda.so
/usr/local/cuda-12.3/targets/x86_64-linux/lib/libcudart.so

+++++++++++++++ WORKING DIRECTORY CUDA PATHS +++++++++++++++

++++++++++++++++++ LD_LIBRARY CUDA PATHS +++++++++++++++++++
++++++++++ /usr/local/cuda-12.3/lib64 CUDA PATHS +++++++++++

++++++++++++++++++++++++++ OTHER +++++++++++++++++++++++++++
COMPILED_WITH_CUDA = True
COMPUTE_CAPABILITIES_PER_GPU = ['7.5', '7.5']
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++ DEBUG INFO END ++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Running a quick check that:
    + library is importable
    + CUDA function is callable

WARNING: Please be sure to sanitize sensible info from any such env vars!

SUCCESS!
Installation was successful!

@dannstrom @WouterMulders1999 Maybe you've solved it?
Thanks
Tomek

@TomekPro unfortunately I haven't. I use ubuntu 20.04 on WSL2 and Jupyter Notebook and it works fine. But I rather use VScode and there it's still a problem. So if you figure it out please let us now :)

Hello,
I also I have this issue with "AttributeError: module 'bitsandbytes' has no attribute 'nn'"

Me too, @alirezamborhan.

Hello, I too have this issue. I use Fedora, and I tried to run a LLM on int4 precision.

https://blog.csdn.net/dandandancpop/article/details/135290871
@alirezamborhan, @matthiasdroth, @Rubiel1
I solved the problem with this tutorial

Hey all, can it be that you have a bitsandbytes directory in your current working directory? That would explain it. We should probably migrate to a src based layout to avoid such issues.
Please run this script and paste the output here.
import os
import bitsandbytes

# Get the directory of the bitsandbytes module
module_dir = os.path.dirname(bitsandbytes.__file__)

# Print the location of the bitsandbytes module
print(f"bitsandbytes module location: {module_dir}")

# List the contents of the module directory
print(f"Contents of {module_dir}:")
for item in os.listdir(module_dir):
    print(item)

I have the exact same issue, I installed 0.44 dev from source, and then ran into this when loading lora weights on top of BLIP-2.  @Titus-von-Koeller when I try to run the above code, I get an error on the first line. What action should be taken based on the output? And also, I tried moving my bitsandbytes source code into a separate directory from my working directory in colab, and it didn't have any effect.


fix: Moved the proxyplugin creation to BeforeAll section

Description
When running selected test with label build-templates-e2e it was failing while deleting ProxyPlugin in the AfterAll block, since we were creating ProxyPlugin in the It block. Moved it to BeforeAll so that the running tests with build-templates-e2e won't fail.
Issue ticket number and link
Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
Checklist:

[ ] I have performed a self-review of my code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added meaningful description with JIRA/GitHub issue key(if applicable), for example HASSuiteDescribe("STONE-123456789 devfile source")
[ ] I have updated labels (if needed)

/test all

/test redhat-appstudio-e2e


UP Marketing Material

As part of the Q3 roadmap our goal is to understanding what the marketing needs are for releasing the UP and have a strategy/plan for what materials we want to make or upgrade for the following quarter.

Website pages have been updated to remove covid specific language and messaging updated to talk about the product as an intermediary for all conditions. Core assets we want to have ready for launch have been identified.

UP one pager to generate interest in flu/rsv and more conditions here: https://docs.google.com/document/d/1hsUbSiW-uIg8COUn0qGwfV96CvyvFe-gkSvnZoq2GUE/edit?usp=sharing


Documentation broken link

In the post https://code.visualstudio.com/blogs/2016/11/15/formatters-best-practices the link in the TIP section that goes to registerDocumentFormattingEditProvider should be this, I believe: https://github.com/microsoft/vscode/blob/a80f904d4e9afa7a05ae57a0123c11b727097129/src/vs/vscode.d.ts#L10089
https://github.com/microsoft/vscode/blob/a80f904d4e9afa7a05ae57a0123c11b727097129/src/vs/vscode.d.ts#L10089
(see https://twitter.com/mcdado/status/1268134981991268358?s=21)

Should it be moved to https://github.com/Microsoft/vscode-docs ?


Pure python implementation of Newton coordinate descent solver

The multinomial gradient and hessian is currently not yet implemented, but I have tested that it works for all other distr.
My %timeit benchmarks suggest that it is 3x-10x slower than 'batch-gradient'.
@jasmainak @raghavrv look at the new methods:
_cdfast(), _gradhess_logloss_1d()
The goal is to translate these to cython or wrap them correctly with numba.
I have also added a cheatsheet.pdf in the latex folder. Have a look there in case of bugs in my code.
Enjoy your liberty and equality!

Coverage decreased (-13.3%) to 80.696% when pulling de372222f4f15763d95d34799a732e12b58b852b on cdfast into ef662c7ce01751209b4094074f6ea9694ab0bfa0 on master.

Coverage decreased (-13.3%) to 80.696% when pulling de372222f4f15763d95d34799a732e12b58b852b on cdfast into ef662c7ce01751209b4094074f6ea9694ab0bfa0 on master.

Coverage increased (+1.0%) to 94.937% when pulling 750d6a20cb203261e17d5f2709a00ec62a37dfed on cdfast into ef662c7ce01751209b4094074f6ea9694ab0bfa0 on master.

The new tests suggest that nothing breaks with the cdfast solver. I have also verified with all distr that the results are comparable to the batch-gradient solver.
I will merge this PR and then start hacking the cython version in a new PR.