Allow the usage of external pci.ids files.

Many systems come with or often have the pci.ids file in a common location.
Allowing the file to be read from said common location, or from any external location for that matter, may be logical.

Are we referring to parsing this file at run-time or compile-time?

Runtime a-la libpci. It would require a fair amount of extra parsing code but it could be worth it for certain use cases.

Another advantage is that it would allow developers to make tools similar to lspci without recompiling them every time a pci.ids update comes out.

Good idea. If we choose to go that route, we can give the user the option to compile in the pci.ids file using include_bytes! or provide their own file path at compile time via an environment variable, and the file gets referenced at runtime.

I should add that if we do this right, we won't actually have to introduce any breaking changes to the API. So this doesn't necessarily have to be done prior to version 1, unless you see any major flaws in the implementation that might have to change if we alter the way we parse the data.


Open-source the code for the other baselines

Could you share the code for the other baselines you compared? I would be grateful if you could make it available. I am looking forward to your reply.

The baselines is already included in our code, encompassing zero-shot, few-shot, and SelfICL. CoT were implemented in the SelfICL repository (https://github.com/ntunlplab/Self-ICL).

I'm very sorry to bother you, but we couldn't find all the baselines, including zero-shot, few-shot, and SelfICL in your code. Could you kindly point out where the baseline code is located in your repository?

For mmlu, https://github.com/yisunlp/DAIL/blob/main/MMLU/Inferencers.py.
For bbh, the results of zero-shot and SelfICL is copied from https://arxiv.org/pdf/2305.15035. For the results not included, we reproduce it with the script in https://github.com/ntunlplab/Self-ICL.

Thank you very much for your patient reply. Best Wishes!

In https://arxiv.org/pdf/2305.15035, they reported the results of zero-shot and SelfICL with gpt-3.5-turbo-instruct, we reproduce the result of it with few-shot and CoT.
For gpt-4-1106-preview, you may need to make some brief modifications to his code repository, mainly in the logic of obtaining responses. Sometimes gpt-4-1106-preview may have some content reviews that prevent it from returning. We have a simple demo that you can refer to for reference, which we use it to get the response of gpt-4-1106-preview in our experiment.

def get_response_from_api(params, query):
if "gpt-4" in params["model"]:
response = None
while response is None:
try:
response = openai.ChatCompletion.create(messages=[{"role": "user", "content": query}], **params)
except:
time.sleep(10)
print("get response failed")
trying = 0
while not response.choices and trying < 5:
trying += 1
print("failed to get content, retrying {}".format(trying))
time.sleep(60)
response = openai.ChatCompletion.create(messages=[{"role": "user", "content": query}], **params)
if trying >= 5:
return False, "No Result"
return True, response.choices[0].message.content
else:
response = None
while response is None:
try:
response = openai.Completion.create(prompt=query, **params)
except:
time.sleep(10)
print("get response failed")
trying = 0
while ("choices" not in response.keys() or not response.choices) and trying < 5:
trying += 1
print("failed to get content, retrying {}".format(trying))
time.sleep(60)
response = openai.Completion.create(prompt=query, **params)
if trying >= 5:
return False, "No Result"
return True, response.choices[0].text.strip()

def get_response_from_api(params, query):     if "gpt-4" in params["model"]:         response = None         while response is None:             try:                 response = openai.ChatCompletion.create(messages=[{"role": "user", "content": query}], **params)             except:                 time.sleep(10)                 print("get response failed")         trying = 0         while not response.choices and trying < 5:             trying += 1             print("failed to get content, retrying {}".format(trying))             time.sleep(60)             response = openai.ChatCompletion.create(messages=[{"role": "user", "content": query}], **params)         if trying >= 5:             return False, "No Result"         return True, response.choices[0].message.content     else:         response = None         while response is None:             try:                 response = openai.Completion.create(prompt=query, **params)             except:                 time.sleep(10)                 print("get response failed")         trying = 0         while ("choices" not in response.keys() or not response.choices) and trying < 5:             trying += 1             print("failed to get content, retrying {}".format(trying))             time.sleep(60)             response = openai.Completion.create(prompt=query, **params)         if trying >= 5:             return False, "No Result"         return True, response.choices[0].text.strip()

def get_response_from_api(params, query):
if "gpt-4" in params["model"]:
response = None
while response is None:
try:
response = openai.ChatCompletion.create(messages=[{"role": "user", "content": query}], **params)
except:
time.sleep(10)
print("get response failed")
trying = 0
while not response.choices and trying < 5:
trying += 1
print("failed to get content, retrying {}".format(trying))
time.sleep(60)
response = openai.ChatCompletion.create(messages=[{"role": "user", "content": query}], **params)
if trying >= 5:
return False, "No Result"
return True, response.choices[0].message.content
else:
response = None
while response is None:
try:
response = openai.Completion.create(prompt=query, **params)
except:
time.sleep(10)
print("get response failed")
trying = 0
while ("choices" not in response.keys() or not response.choices) and trying < 5:
trying += 1
print("failed to get content, retrying {}".format(trying))
time.sleep(60)
response = openai.Completion.create(prompt=query, **params)
if trying >= 5:
return False, "No Result"
return True, response.choices[0].text.strip()


MaxAmountOfEnchantmentsToggle does not work

My players try to add one custom enchantment and it does not work. It says you hit the max limit and I don't have include vanilla enchantments on.
  EnchantmentOptions: #Options for enchanted books.     MaxAmountOfEnchantmentsToggle: True #Turn on an off the Max amount of enchantments     IncludeVanillaEnchantments: False #Count Vanilla enchantments also?     MaxRageLevel: 10 #The max amount rage can stack when fighting.     Right-Click-Book-Description: True #Toggles wether when you right click the enchantment book if it messages the player the description or not.

You either turn it off, or you give players the CrazyEnchantments.Limit.# permission to allow them to put more enchantments on the armour.


Housekeeping

What was wrong?

Circle wasn't running backend tests
Readme quickstart didn't have enough info
Travis used old pypy3 container

How was it fixed?

Added backend tests to circleci
Flesh out quickstart with backend, example hash, and docs link
Upgrade Travis pypy3 container

Cute Animal Picture

Closing to re-open to trigger circle


Feature: 3D print or sewing partnership

I think it would be very nice if you could order your own custom Gopher to be 3d-printed or sewn specially for you. I know I would!

@hazcod me too - but how on earth do we do that? :)


Blade doesn't parse the @laravelPWA directive

`<!doctype html>
Laravel - @yield('title')

@laravelPWA
@yield('styles')
@livewireStyles
@yield('content')
@yield('scripts')
@livewireScripts`
@laravelPWA not working. ! Any solution ?

Do you have a repo where i can reproduce that issue? :)

This is a duplicate of #58


document current tenant attach API semantics

We currently return 202 as soon as the tenant is allocated in memory before we've written out the marker file. So, the /attach API currently does not have a transactional character. For example, it can happen that we respond with a 202 and then crash before writing out the marker file. In such a case, it is important that the client

observes the lost attach (by polling tenant status and observing 404)
and consequently retries the attach.

It has to do it in this loop until it observes the tenant as "Active" in the tenant status.  If the client doesn't follow this protocol and instead goes to another pageserver to attach the tenant, we risk a split-brain situation where both the first and second pageserver write to the tenant's S3 state.
The improved description highlights the consequences of this behavior for clients that use the /attach endpoint.
My understanding is that the tenant relocation logic being worked on in cloud#4740 implements retries, but only until it gets a 202. It will need to implement the polling after the 202 to be correct.
The motivation for this write-up is that, in a future PR, we'll extend the /attach endpoint with an option to provide the tenant config. If we decide to leave the non-transactional behavior of /attach unmodified, we will be able to avoid persisting the tenant config. Conversely, if we decide that the /attach API should become transactional, we'll need to persist the tenant config in the attach-marker-file before acknowledging receipt of the /attach operation.
refs https://github.com/neondatabase/cloud/pull/4740 refs https://github.com/neondatabase/neon/issues/2238 refs https://github.com/neondatabase/neon/issues/1555

if I read the cloud code right, it will wait after the attach operation for the pageserver to answer with the tenant status (see goapp/internal/operationsv2/operation_executor.go line 766 before the cloud change and  line 735 in @NanoBjorn change)


.to_tf_dataset returns an empty object

Everything seems to be in place before transforming dataset to tf format.

I am not quite sure what went wrong, but the output is empty:

Final error message running model.fit(...):

Any advice is much appreciated. Thank you!

Which versions are you using?

Thank you for your reply!
tensoflow 2.6.0
keras 2.6.0
huggingface-hub 0.4.0

and transformers/datasets?

Oh, of course, sorry.
datasets 2.0.0
transformers 4.16.2


Provide analog of praw.objects.Submission.replace_more_comments for praw.objects.Comment

Currently, it's possible to automagically traverse the comment tree of a submission and replace the MoreComment instances with the actual comments they represent. However, it isn't possible to do the same thing with the reply tree of an individual comment.

Great idea.

Added in https://github.com/praw-dev/praw/commit/1e0e6684b7f3383d41123319a243402fa87aec2d.
Please let me know if you experience any issues with this update as it's the biggest change since the PRAW 4.0.0 release.


Questions about Value_Tensor_map and Evaluated_Value_map? (Not an issue, just try to understand them...)

I have just gone through TRTorch's 2020 GTC talk/slides/documentation focusing mainly on the graph conversion implementation part. There are some confusions of concepts and questions:

What's the relationship between torch::jit::Values and torch::jit::IValue, Are they the same thing? I noticed they are used interchangeably in some situations and are referring to different classes in others.
Why do we need to record Value -> ITensor map and Value->IValue map? What's the main use of these two maps?

Could someone help me? Thanks in advance!

(Moving this to discussions so it is easier to find later)
To answer 1.
torch::jit::Values are abstract representations of data flow in the PyTorch graph.

torch::jit::IValues contain realized data which corresponds to the values in the graph. In the JIT interpreter these IValues are put into a stack machine to feed node executors.

So in this case the graph says Values %0 and %1 are just some int, then when executed those Values are represented in the stack by IValues containing 2 and 4, the consumer of %2 will take 6 later in the execution.
To answer 2:
Nodes dont have attributes or anything like ONNX, they are mostly purely functional. All configurations are inputs to the node. So that for say aten::max_pool2d, you would get inputs like this
aten::max_pool2d(Tensor self, int[2] kernel_size, int[2] stride=[], int[2] padding=[0, 0], int[2] dilation=[1, 1], bool ceil_mode=False) -> (Tensor)

So in the compiler we need the realized data (IValues) since TensorRT engines are built ahead of time. The role of evaluators is to calculate these IValues. So kernel size, stride, padding, dilation and ceil mode will all be calculated during the course of compilation. We store the IValues in evaluated_value_map keyed to the torch::jit::Value it would realize in the JIT interpreter because the JIT graph is SSA (Static single assignment) and this new IValue might be used in multiple places.
In TensorRT, the only dataflow is the actual Tensors being operated on, they are represented by nvinfer1::ITensors in the TRT graph.

In the case of aten::max_pool2d the input tensor self and the output tensor need to be represented in the TRT graph. Converters are responsible for creating TRT layers, so they take the input ITensor created by a different layer and all of the IValues calculated by the evaluators to create the new layer. In that process they associate the output JIT Value with the corresponding output TRT ITensor in the value_tensor_map.
So as we traverse the JIT graph during compilation we know that all preceding JIT Values will have either an IValue in the evaluated_value_map or an ITensor in the value_tensor_map. So when we hit max_pool2d or any other op all we need to do is use the input Values to that node to look up in these maps all of the data we need to create the next layer.
If you look at the documentation for how to write a converter you can see how that is used.
https://nvidia.github.io/TRTorch/contributors/writing_converters.html

(Moving this to discussions so it is easier to find later)
To answer 1.
torch::jit::Values are abstract representations of data flow in the PyTorch graph.

torch::jit::IValues contain realized data which corresponds to the values in the graph. In the JIT interpreter these IValues are put into a stack machine to feed node executors.

So in this case the graph says Values %0 and %1 are just some int, then when executed those Values are represented in the stack by IValues containing 2 and 4, the consumer of %2 will take 6 later in the execution.
To answer 2:
Nodes dont have attributes or anything like ONNX, they are mostly purely functional. All configurations are inputs to the node. So that for say aten::max_pool2d, you would get inputs like this
aten::max_pool2d(Tensor self, int[2] kernel_size, int[2] stride=[], int[2] padding=[0, 0], int[2] dilation=[1, 1], bool ceil_mode=False) -> (Tensor)

So in the compiler we need the realized data (IValues) since TensorRT engines are built ahead of time. The role of evaluators is to calculate these IValues. So kernel size, stride, padding, dilation and ceil mode will all be calculated during the course of compilation. We store the IValues in evaluated_value_map keyed to the torch::jit::Value it would realize in the JIT interpreter because the JIT graph is SSA (Static single assignment) and this new IValue might be used in multiple places.
In TensorRT, the only dataflow is the actual Tensors being operated on, they are represented by nvinfer1::ITensors in the TRT graph.

In the case of aten::max_pool2d the input tensor self and the output tensor need to be represented in the TRT graph. Converters are responsible for creating TRT layers, so they take the input ITensor created by a different layer and all of the IValues calculated by the evaluators to create the new layer. In that process they associate the output JIT Value with the corresponding output TRT ITensor in the value_tensor_map.
So as we traverse the JIT graph during compilation we know that all preceding JIT Values will have either an IValue in the evaluated_value_map or an ITensor in the value_tensor_map. So when we hit max_pool2d or any other op all we need to do is use the input Values to that node to look up in these maps all of the data we need to create the next layer.
If you look at the documentation for how to write a converter you can see how that is used.
https://nvidia.github.io/TRTorch/contributors/writing_converters.html


[ast] Make ast a comportable module or at least a comportable front-end

The fact that ast is a "comportable-like" module that exists outside of the top_earlgrey boundary causes a lot of issues that frankly could be otherwise avoided.
In the future, there should not be any comportable-like module that exists outside of the boundary of top_earlgrey.
If there is a proprietary portion that needs to be hidden, there are two solutions

Follow the example of otp / flash and wrap the proprietary portion under the comportable open source.
Maintain the comportable open source front end inside earlgrey, and forward a struct connecting that module to some entity sitting outside top_earlgrey.  The outside world SHOULD NOT have any concept of comportable buses / clocks / alerts etc.

It may be too late to do this now, so we'll have to live with the mess we made and fix it some other day.

@arnonsha
@zi-v
@Jacob-Levy
@dsandrs

Totally agree. Ideally, AST would be a comportable IP (or the AST open-source wrapper for that matter).

My vote is for standardizing option 1.

We discussed this in one of the latest sync meetings and concluded to keep AST as is.
Note that the AST register space is opaque to software, During boot, the ROM code copies a configuration blob from OTP to AST. The ROM code is not aware of the blob content, the chip creator sets the right values into the relevant OTP space during production flow. After the AST configuration is done, AST registers will be blocked.

Several files inside hw/top_earlgrey/ip/ast/rtl claim to be auto-generated (PLEASE DO NOT HAND-EDIT THIS FILE. IT HAS BEEN AUTO-GENERATED in the header).
However, I cannot find any generator for these files or ast in general. Did I miss something or does the header not apply?

Afaik, there is no opensource generator available for this.
But Nuvoton might have generated that internally before upstreaming.
@zi-v @meisnere can these comments be removed if there is no opensource generator?

The generator is closed source. I prefer to leave the comment there, so no one will try to change it manually by mistake.

Thanks for clearing this up. Maybe it would make sense to include the fact that the generator is not open-sourced in said comments?
During the creation of a customized toplevel, this left me a bit puzzled as I could not find such a generator in the repository. In the end I copied the respective files from top_earlgrey, along with some other .core and HDL files for various top-level modules (rstmgr, clkmgr, pwrmgr, sensor_ctrl) that topgen.py does not generate automatically.

Just to be clear, the only issue pending in this is to extend the comment to state the generator for this file is not open-source to avoid open-source developers trying in vain to find the generator.

Just to be clear, the only issue pending in this is to extend the comment to state the generator for this file is not open-source to avoid open-source developers trying in vain to find the generator.

Closing as completed in #23356.


v2.1.2

Checklist

[X] Used a personal fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[X] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[X] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.
Here's what I've got...
For recipe:

requirements: host: python>=3.6 must contain a space between the name and the pin, i.e. python >=3.6
requirements: run: python>3.6 must contain a space between the name and the pin, i.e. python >3.6
noarch: python recipes are required to have a lower bound on the python version. Typically this means putting python >=3.6 in both host and run but you should check upstream for the package's Python compatibility.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@conda-forge-admin, please rerender


[Web support] Member not found: 'InAppWebViewController.setWebContentsDebuggingEnabled'

I'm aware the web support is not yet announced, but I'm testing the developer branch, so I guess I could submit this isssue.
Environment

Technology
Version

Flutter version
3

Plugin version
developer branch

Android version

iOS version

Xcode version

Device information:
Description
Expected behavior:
Current behavior:
Steps to reproduce
This line of code doesn't compile for web :
await InAppWebViewController.setWebContentsDebuggingEnabled(true);
Images

Stacktrace/Logcat

The new version 6.0.0 has been released now!
You can't use methods that aren't supported on the Web platform!
Check the code docs if the method you want to use is supported or not on web and use the kIsWeb flutter constant, for example:
if (!kIsWeb && defaultTargetPlatform == TargetPlatform.android) {
    await InAppWebViewController.setWebContentsDebuggingEnabled(true);
}

// ...

InAppLocalhostServer localhostServer = new InAppLocalhostServer(documentRoot: 'assets');
if (!kIsWeb) {
    await localhostServer.start();
}


Changed DRIVEDOCS.1.3v0.1 to SHALL per RFC Comment 178

🗣 Description
Changed DRIVEDOCS.1.3v0.1 from a SHOULD to a SHALL.
Fixes #178
💭 Motivation and context
Came from an RFC Comment

🧪 Testing

✅ Pre-approval checklist

[ ] This PR has an informative and human-readable title.
[ ] Changes are limited to a single goal - eschew scope creep!
[ ] If applicable, All future TODOs are captured in issues, which are referenced in the PR description.
[ ] The relevant issues PR resolves are linked preferably via closing keywords.
[ ] All relevant type-of-change labels have been added.
[ ] I have read and agree to the CONTRIBUTING.md document.
[ ] These code changes follow cisagov code standards.
[ ] All relevant repo and/or project documentation has been updated to reflect the changes in this PR.
[ ] Tests have been added and/or modified to cover the changes in this PR.
[ ] All new and existing tests pass.

✅ Pre-merge Checklist

[ ] This PR has been smoke tested to ensure main is in a functional state when this PR is merged.
[ ] Squash all commits into one PR level commit using the Squash and merge button.

✅ Post-merge Checklist

[ ] Delete the branch to clean up.
[ ] Close issues resolved by this PR if the closing keywords did not activate.

The Last Modified date was fixed.


Add coveralls for code coverage

Hooray for promoting test coverage!
I submitted this because I was interested in learning about setting up code coverage tools. Also I wanted to know whether the test cases for my rules are covering every branch of the code.
Feel free to close this PR if there are reasons to not include a code coverage tool. If folks are interested in merging this, then @Snugug (or someone with admin access) will have to set up coveralls.io to track this repo.
Additionally, if this gets merged we should add the coveralls badge to the readme.
DCO 1.1 Signed-off-by: Ben Rothman<EMAIL_ADDRESS>

@benthemonkey thanks for this but could we open up issues before starting these types of PR please. Gives everyone a chance to discuss it or give any info on other plans we may have had. I'll leave this one for @Snugug

OK will do next time.

I just looked over the language I used in this PR. I updated it to not be as assuming, given that I didn't ask permission before submitting this PR. :smile:

@DanPurdy @Snugug This has been sitting for awhile. I don't mind closing it, but I'd like to hear back about it, when you get the chance.

We're busy!
I'm OK merging this in as long as you update to the latest develop (currently can't be merged).
We can debate the merits of code coverage elsewhere

Done. I'm not trying to be pushy. I know you were away doing exciting things. :smile:

:P


Proposal: add deisctl ssh

All too often I've needed to ssh into the cluster and inspect a unit. It would be great if we could call fleetctl ssh in the background such that I do not require fleetctl on my local host to manage my containers, similar to how we call fleetctl list-units with deisctl list.

+1 on this, would really be helpful to have this feature


Unable to download webpage: HTTP Error 403: Forbidden - learning.oreilly.com

Checklist

[ X ] I'm reporting a broken site support
[ X ] I've verified that I'm running youtube-dl version 2019.05.20
[ X ] I've checked that all provided URLs are alive and playable in a browser
[ X ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[ X ] I've searched the bugtracker for similar issues including closed ones

Verbose log

/home/alex/.local/bin/youtube-dl -v -n --sleep-interval 60 --write-sub --convert-subs srt --playlist-reverse --skip-unavailable-fragments -ic --socket-timeout 5 --retries infinite --write-description --write-info-json --write-annotations -o '%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s' https://learning.oreilly.com/library/view/go-in-24/9780134771922/
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '-n', '--sleep-interval', '60', '--write-sub', '--convert-subs', 'srt', '--playlist-reverse', '--skip-unavailable-fragments', '-ic', '--socket-timeout', '5', '--retries', 'infinite', '--write-description', '--write-info-json', '--write-annotations', '-o', '%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s', 'https://learning.oreilly.com/library/view/go-in-24/9780134771922/']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.05.20
[debug] Python version 3.4.2 (CPython) - Linux-4.4.38-v7+-armv7l-with-debian-8.0
[debug] exe versions: avconv 11.12-6, avprobe 11.12-6
[debug] Proxy map: {}
[safari:course] Downloading login page
[safari:course] Logging in
[safari:course] Completing login
ERROR: Unable to download webpage: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/home/alex/.local/lib/python3.4/site-packages/youtube_dl/extractor/common.py", line 626, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/home/alex/.local/lib/python3.4/site-packages/youtube_dl/YoutubeDL.py", line 2227, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.4/urllib/request.py", line 461, in open
    response = meth(req, response)
  File "/usr/lib/python3.4/urllib/request.py", line 571, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.4/urllib/request.py", line 499, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 433, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 579, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

Description

Just tried to download this  O'reilly courses and I get always the same error.
The download work without any problem with this url: https://learning.oreilly.com/videos/clean-code/9780134661742

For any work on this you must provide reproducible environment.

What is missed here?


Support SDP Transition

The docs engineering team is finishing up with the improvements in their system that they call SDP.
Essentially, it boils down to improvements to publishing speed:

Adjusted yml artifact generation that reduces that amount of files that need to be copied during publishing
Changes to allow parallel processing of the docfx file, which significantly reduces publishing time

The azure-sdk docs CI jobs all re-use the same processing, but are different in how we consume the CI targeting.  Due to that, the docs devs are filing PRs with the necessary adjustments.
However, we need to make certain that we adjust CI to actually continue generating the proper yml.
This needs to happen for each repo.

[ ] MicrosoftDocs/azure-docs-sdk-python
[x] MicrosoftDocs/azure-docs-sdk-node
[ ] Azure/azure-docs-sdk-java
[ ] Azure/azure-docs-sdk-dotnet

Finished with testing last night. Azure-docs-sdk-node is now SDP conformant.

Python and Java are swapped. Python merged to master. Java is swapped to leveraging the Java-Merf branch.


scaling sidebar independantly

my problem is when I have my OSRS client and HUD scaled properly for full screen. the Runelite side bar is very small.
If i change windows DPI scaling both the OSRS game and Runelite scale together which effects the smoothness factor of the game.
the solution for this would be adding scaled side bar. i would say just resize text but that may cause overlapping.

this is approximately what graphics/HUD scaling i am after, but if i want to scale the side bar i have to scale everything.  which takes away from other elements.

There is no way to properly scale just sidebar. DPI scaling works good enough.

I disagree with "DPI scaling works good enough".
I don't want to turn on a 125-150% DPI scaling across my entire monitor just because I find the RuneLite UI difficult to read because of tiny font choices. All of my other applications are perfectly readable without any issues at 100%.
Windows currently doesn't allow to increase the DPI scaling for 1 single app. You can set a higher DPI for everything and then turn it off per app, but that's not a solution when it's just one single app that is hard to read.
I've tried passing in the scaling factor to the JAR, that works, technically. But it doesn't improve readability at all, if anything it makes it even worse because the font doesn't scale well.

I'll have to agree with @T-Grave  it's a bit cumbersome to have to set the DPI-scaling system-wide for practically only RuneLite. The best solution would of course be to being able to scale it independently of the game itself.
I think a good compromise is allowing us to pass jvm/scaling arguments which there exists a PR already for here https://github.com/runelite/launcher/pull/19 , although it seems forgotten or otherwise.

I'll have to agree with @T-Grave it's a bit cumbersome to have to set the DPI-scaling system-wide for practically only RuneLite. The best solution would of course be to being able to scale it independently of the game itself.
I think a good compromise is allowing us to pass jvm/scaling arguments which there exists a PR already for here runelite/launcher#19 , although it seems forgotten or otherwise.

The launcher has two modes of operation, one will launch the client in the same VM as the launcher is in, the other will launch a new VM.  For the former, which is the most common way it operates, we cannot adjust the JVM arguments in the launcher as the launcher runs post-VM startup.  It does this so that Discord can identify it by its process name RuneLite.exe. So it is not as simple as it seems.

@Adam-

The launcher has two modes of operation, one will launch the client in the same VM as the launcher is in, the other will launch a new VM. For the former, which is the most common way it operates, we cannot adjust the JVM arguments in the launcher as the launcher runs post-VM startup. It does this so that Discord can identify it by its process name RuneLite.exe. So it is not as simple as it seems.

Isn't that the point of -jvmargs to be able to pass jvm arguments through the chain? Or am I mistaken?

The code processing the -jvmargs argument is Java code, so by then it is too late since the JVM must already be running.

Has there been any solution for this problem that doesn't involve scaling my whole monitor? RuneLite is the only application causing eye strain trying to read the tiny text in the sidebar. @ironicalbob @deathbeam

Almost 2 years later, is there a possibility to scale just the RuneLite UI? I know there is DPI Scaling but this is not suitable for everyone.


Update the contributing page about the GEP process

/kind cleanup
/kind documentation
What this PR does / why we need it:
While updating GEP-1742 to experimental (#2392), I noticed that we have some redundant info about the new GEP process in the contributing section of the site. This PR updates that page to refer to the correct place (/geps/overview), so we only have one place to update.
Does this PR introduce a user-facing change?:
NONE

Thanks @youngnick!
/lgtm

/hold cancel


Refactored docs to create a getting started, user guide and contributing guide

This PR refactors the existing documentation in docs/source.
Earlier docs relied on presenting the README file and a list of notebooks.
New structure has four main sections:

Getting Started: Intro to dowhy, installation, etc. (Relevant for a new user)
User guide: A detailed walk-through of using dowhy
Case studies using DoWhy
Contributing: Guide for contributors

In addition, it has the GCM section which can be integrated within the User Guide in a future PR.
No new content has been added. It is merely a restructuring of the existing content. But this structure will make it easy for anyone to add more content.
In particular, the files are organized according to the sections above (getting_started, user_guide) etc. to make it easy to add content.
Note:  The docs/source/getting_started/install.rst is empty because its content is already in PR #473 . Once this PR is merged, we can merge 473 too.
Result: The right sidebar looks much cleaner.

Love it! Couple of questions:

Given the exhaustive scope of Tutorial on Causal Inference and its Connections to Machine Learning (Using DoWhy+EconML), would it make sense to move it to the examples? Or is your plan to eventually integrate it into the user guide? It seems quite large for the Getting started section. No strong opinion though.
I guess the moving the gcm user guide into the main user guide would happen as a separate step, right? (Same for the gcm examples)
There seems to be quite some duplication between the docs and the README. Do you have a plan for how to handle this?

Ah yes, the tutorial is probably too much for Getting Started. I didn't want to create a separate section just for the tutorial. Maybe we can create a section "Tutorials/Case Studies" and add this notebook there along with the two case studies? what do you think?
Yes, I was thinking that once we merge this, you can probably raise a PR on how best to merge the GCM user guide and examples.
That's a good question. I'm not sure what's a good solution. We'd definitely want to avoid duplicate text floating around. Here's one thought: Going forward, the github readme can focus more on code examples and less on the conceptual stuff that's currently in there. In terms of code examples, right now the readme only includes the most basic ones, but dowhy has added many more features since then. So one option is to focus the readme on "what's possible/different features of dowhy", something like EconML's readme. I'm not 100% sure though. Thoughts?

Ah yes, the tutorial is probably too much for Getting Started. I didn't want to create a separate section just for the tutorial. Maybe we can create a section "Tutorials/Case Studies" and add this notebook there along with the two case studies? what do you think?

Yes, sounds good to me.

Yes, I was thinking that once we merge this, you can probably raise a PR on how best to merge the GCM user guide and examples.

👍

That's a good question. I'm not sure what's a good solution. We'd definitely want to avoid duplicate text floating around. Here's one thought: Going forward, the github readme can focus more on code examples and less on the conceptual stuff that's currently in there. In terms of code examples, right now the readme only includes the most basic ones, but dowhy has added many more features since then. So one option is to focus the readme on "what's possible/different features of dowhy", something like EconML's readme. I'm not 100% sure though. Thoughts?

There's always going to be some duplication and I think that's fine. Some big projects like numpy reduce this duplication to the absolute minimum and refer directly to the docs, whereas other big projects like PyTorch have lots of content already in the README, therefore lots of duplication, but with additional cross-links to documentation. I think we can find our sweet spot somewhere in the middle.
Given that we're reworking the documentation right now, I would consider just keeping the most relevant stuff in the README, i.e. not too many examples, but just basic ones. Similar to the getting started section in the docs. In fact, I wouldn't mind if the README resembles the Getting Started section to a certain extent, because as a first time visitor I like when I already get everything necessary from the README to get started.
Roughly, this means, we include:

Introduction / An end-to-end library for causal inference

News?
The need for causal inference
Sample causal inference analysis in DoWhy
GCM-based inference (experimental)

Installation
Comparison to other causal inference packages
Citing this package
Roadmap?
Contributing

No strong opinions on order or which sections exactly to include. But we should probably have some sections that explicitly links to the documentation and specifically to examples in the docs.
What do you think?

Great, I made the change for tutorial, moving it to tutorials/case studies.
Like your suggestion on the README. I can work on a draft PR for that, and we can aim to merge it simultaneously with the v0.8 release, when these new docs will be available on the website.


update init container image version to 2.2.1 from 2.2.0-1

Update the confluent-init-container from 2.2.0-1 to 2.2.1

@ptyagii could you update your fork branch with latest and resolve the conflicts? We can then merge. Thanks!

@rohit2b sure, resolved the conflicts.


Minor correction to how we return error

An earlier commit made a change to return Error objects instead of error strings.
However, it caused a regression as it did not check if errorMessage is empty before returning an Error instance.
With this change, an Error object will be returned only if the error string is non-empty.

@danderson00


Shopy themekit "theme watch" not detecting the file changes

I have installed the new themekit for the Shopify theme development using choco.
OS Name:                   Microsoft Windows 10 Pro
OS Version:                10.0.19041 N/A Build 19041
System Type:              x64-based PC
ThemeKit 1.1.1 windows/amd64
When I am using theme watch and changing files it's being stuck to below state. Nothing happening. No file is being updated.
An update for Themekit is available. To update please run theme update
20:01:31 [development] Tenacious Pets: Watching for file changes to theme<PHONE_NUMBER>71

Please help me with this note.

add :  theme watch --allow-live flag


NameError found flax repo due to case sensitivity

when I was executing fine tuning tutorial notebook I found the code error in this notebook : https://github.com/google-deepmind/gemma/blob/main/colabs/fine_tuning_tutorial.ipynb

And also getting the same error in the below notebooks, could you please recheck it
https://github.com/google-deepmind/gemma/blob/main/colabs/gsm8k_eval.ipynb
https://github.com/google-deepmind/gemma/blob/main/colabs/sampling_tutorial.ipynb
I am providing a gist for reference .

Hi @KumarGitesh2024,
I was able to reproduce the issue, and to prevent this error, it's necessary to change the variant to VARIANT. Additionally, some users have already raised pull requests to address this problem. Our team will review and provide feedback as soon as possible.
Thank you.

Hi @KumarGitesh2024,
Could you please confirm if this issue is resolved for you with the above comment ? Please feel free to close the issue if it is resolved ?
Thank you.

Closing this issue as stale. Feel free to re-open if the issue still persists. Thank you!


Vks4git/protostar

Ready to approve, too, but the fact that proving protocol has to use Symbolic DSL API bothers me, will think about it when I get back from my vacation

Ready to approve, too, but the fact that proving protocol has to use Symbolic DSL API bothers me, will think about it when I get back from my vacation

We take one circuit (recursion step) and construct another one (full recursion). But maybe there is a way to decouple it like you suggested with Plonk.

We take one circuit (recursion step) and construct another one (full recursion).

Sounds like something to be exposed via Symbolic API to end users to be used in recursive datatype definitions and recursive functions! Will look into it once I get back from vacation, sounds very interesting.


Family page designed **not done working on state/etc.

Description
I designed a Family page where, as a parent, eventually one could add, change, or delete students on the account. I first designed the page on Figma, then implemented the page based on the design. I edited the following files:

src/components/pages/ParentFamily - I added this folder to hold the ParentFamilyHome component.

src/components/common/Banner.js - I added the path so that the correct title is displayed in the banner.

src/components/pages/ParentFamily/ParentFamilyHome.js - This is the component that most of the work was done in.  I imported the Banner and ParentSidebar to keep formatting consistent between the dashboard and this page. There is no state functionality in this component (if anyone is able to help implement this, please let me know), I only have the styling done as of now.

src/index.js - I imported my new ParentFamilyHome component and created a new route for it.

src/styles/ParentStyles/index.less - I added the styling for the FamilyPage to the bottom of this .less file.

Trello card link
Loom video
Fixes # (issue)
Type of change
Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] This change requires a documentation update

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have removed unnecessary comments/console logs from my code
[ ] I have made corresponding changes to the documentation if necessary (optional)
[x] My changes generate no new warnings
[x] I have checked my code and corrected any misspellings
[x] No duplicate code left within changed files
[x] Size of pull request kept to a minimum
[x] Pull request description clearly describes changes made & motivations for said changes

I am confused about the last commit "syncs package-lock".
Looks good otherwise!

I had merge conflicts with the package-lock.json. So, I had to copy the code from the repo & replace the code in my local version with that code. Then I made the 'sync package-lock' commit & there were no merge conflicts.


How to link peers from peers.json with each other  ?

Hi,  I use stable version.
I've locally setup small 10 peers network (each node have separate folder, config, db and port) , and provided peers.json file:
[["<IP_ADDRESS>","30301"],["<IP_ADDRESS>","30302"],["<IP_ADDRESS>","30303"],["<IP_ADDRESS>","30304"],["<IP_ADDRESS>","30305"],["<IP_ADDRESS>","30306"],["<IP_ADDRESS>","30307"],["<IP_ADDRESS>","30308"],["<IP_ADDRESS>","30309"],["<IP_ADDRESS>","30310"] ]
But when I start nodes, I see this list only as http://<IP_ADDRESS>:30203/peers/known and http://<IP_ADDRESS>:30203/peers/connected is empty.
Also, as I see from logs, each peer started to mine it's one chain separately.
How to link peers with each other and mine common chain?

try do use a central bootstrapping node and let the peers connect to it.
bootstrapping node:
pyeth -l 30300 -d data0/
other nodes
pyeth -r <IP_ADDRESS> -p 30300 -l 30301 -d data1/
pyeth -r <IP_ADDRESS> -p 30300 -l 30302 -d data2/
pyeth -r <IP_ADDRESS> -p 30300 -l 30303 -d data3/
...

and  other peers bootstrap node will share peer with each other?

yes, they all connect to the boot strap node, and then ask frequently for other peers. bootstrap node returns all known (i.e. connected) peers.

Thank you, I've just launched 10 nodes, with one minig (which was master with id 0) and everything was working as expeted, new blocks were mining, chain was synching, but then I've launched transcation generator:
while [ 1 ];
do
pyethclient quicktx b8a1e70b90b28cde1ef7402bc280c5b5a357b499 1 '' 80d6c6da7450f8d84cea853286c00c6458cb7d0d037d63f59da64d264d7bc529 -H <IP_ADDRESS> -p 30202;
done
and after few successfull transactions it started to slowdown. After few blocks chain stops to grow and propofate new block, here is hang state:
node_id, block_number, last_block_hash
9 23 c7c6bdcfc9efb2ca868f9932e0ed7522725622e2ade18432fac5bce6b8aa9d19
8 23 c7c6bdcfc9efb2ca868f9932e0ed7522725622e2ade18432fac5bce6b8aa9d19
7 23 c7c6bdcfc9efb2ca868f9932e0ed7522725622e2ade18432fac5bce6b8aa9d19
6 23 c7c6bdcfc9efb2ca868f9932e0ed7522725622e2ade18432fac5bce6b8aa9d19
5 23 c7c6bdcfc9efb2ca868f9932e0ed7522725622e2ade18432fac5bce6b8aa9d19
4 23 c7c6bdcfc9efb2ca868f9932e0ed7522725622e2ade18432fac5bce6b8aa9d19
3 23 c7c6bdcfc9efb2ca868f9932e0ed7522725622e2ade18432fac5bce6b8aa9d19
2 23 c7c6bdcfc9efb2ca868f9932e0ed7522725622e2ade18432fac5bce6b8aa9d19
1 23 c7c6bdcfc9efb2ca868f9932e0ed7522725622e2ade18432fac5bce6b8aa9d19
0 28 3201d9d6bbe9b2a527cd713b58459fa6485bead493232d92bf4f611616008c4e
and on master  node logs I see many lines of this kind :
....
Thread-708:chainmanager: remote_transactions_received: [<Transaction(ab59)>, <Transaction(8cd8)>, <Transaction(35f4)>, <Transaction(4579)>, <Transaction(99be)>, <Transaction(8354)>, <Transaction(2f91)>, <Transaction(e6c0)>, <Transaction(bd18)>, <Transaction(0275)>, <Transaction(0941)>, <Transaction(5166)>, <Transaction(f5fa)>, <Transaction(cb4e)>, <Transaction(6626)>, <Transaction(4d27)>, <Transaction(ba46)>, <Transaction(1579)>, <Transaction(6a5e)>, <Transaction(82f8)>, <Transaction(ae76)>, <Transaction(a9b4)>, <Transaction(3cdf)>, <Transaction(c7cd)>, <Transaction(680c)>, <Transaction(55ff)>, <Transaction(1bb0)>, <Transaction(a077)>, <Transaction(f9f2)>, <Transaction(b617)>, <Transaction(6e3a)>, <Transaction(40f2)>, <Transaction(68fa)>, <Transaction(5de0)>, <Transaction(c093)>, <Transaction(69af)>, <Transaction(844a)>, <Transaction(0aa4)>, <Transaction(1cb7)>, <Transaction(5c37)>, <Transaction(0ca4)>, <Transaction(3542)>]
Thread-669:chainmanager: add transaction <Transaction(4d27)>
Thread-708:chainmanager: add transaction <Transaction(ab59)>
Thread-829:chainmanager: get_transactions called
Thread-715:chainmanager: add transaction <Transaction(6626)>
.....
What could be wrong here and were to start  debug it ?

what happens if you add a sleep 1 in the while loop


sinogram

Hello,
i'm doing the same simulation as you except for the part "phantoms_meep" where i'm using python instead of C++.
I have several fields acquired with different angles, and i would like to know how did you do to calculate the sinogram. are you recording two fields (with and without phantom) for each angle ?.
Thank you

Yes, exactly. I divide the field with phantom by the field without phantom (background correction).

after that the divsion with/without how did you calculate the sinogram ?

You can find that code here: https://github.com/RI-imaging/FDTD_sinogram/blob/master/meep_tomo/meep_tomo/extract.py


vram usage does not go back down after model unloads

What is the issue?
when i set keep alive to 0 the memory usage does not go all the way back down. also it uses system ram when vram still avalible
gpu 7800xt
platform windows
rocm version 6.1
OS
Windows
GPU
AMD
CPU
AMD
Ollama version
0.4.1

server-3.log
app-4.log

What's the output of ollama ps?

screenshot of before and after closing ollama
and output of ollama ps

here is when a model runs and stops

no if i quit ollama it goes back down, that is ollama using it

i found a work around, for n8n(which is what im mainly using ollama for atm) just tell it to kill ollama and it forces it to dump the memory, but that should not be required

@CraftMaster163 can you check detailed processes to see if ollama_llama_server is still running when you see the memory leakage, or if only the main ollama process is running?  When the model goes idle, we should shut down the ollama_llama_server which should release GPU memory, however we do GPU discovery in the main ollama process.  That code should unload the GPU libraries when not actively querying to avoid additional VRAM usage.
Looking at your driver version, this might be a dup of #7107
Does the behavior disappear if you downgrade the driver to 24.8.1?

yes i just checked and ollama_llama_server is still running, if i close it the vram goes down again

also ollama.exe does not get rid of its vram usage, if i stop it, it restarts fine and goes back down to my normal vram usage

@CraftMaster163 can you clarify if ollama ps shows nothing running while ollama_llama_server is still running?

i assume /bye unloads model

i assume /bye unloads model

That just closes the client, but the model will stay loaded until the timeout expires.  By default this is 5m but you can override.  Can you verify if the VRAM usage drops back down once ollama ps no longer shows a model loaded?
https://github.com/ollama/ollama/blob/main/docs/faq.md#how-do-i-keep-a-model-loaded-in-memory-or-make-it-unload-immediately

hm now it closes, when i removed the commands from n8n, it does stop but ollama.exe does not clear its cache, here is a photo showing the graph going down after closing ollama

hm now it closes, when i removed the commands from n8n, it does stop but ollama.exe does not clear its cache, here is a photo showing the graph going down after closing ollama

Can you clarify?  When ollama ps shows no model loaded, does ollama_llama_server stop and does the VRAM usage drop back down to the baseline you area expecting?  If so, this doesn't sound like a bug.
I'm not familiar with n8n, but if it calls the Ollama API and keeps the model loaded, that might explain what you are seeing.

no i told it to unload model, there is a spike, it unloads the context but does not fully unload model, here is a screenshot of the three spikes. normal usage

after ollama runs with keep alive 0

and after i quit ollama

if i let my discord bot using ollama run for a while it goes to full usage, its just llama3.2:3b which does not need that much memory

https://drive.google.com/file/d/1x_CYMP45GIFcMIAYwTgio3_oYyrUsPNP/view?usp=sharing
here is a video of the issue


XCode 16 throws error message ion Defaults code

Since updating to XCode 16, the following error is thrown in the Defaults package: "Accessing StateObject's object without being installed on a View. This will create a new instance each time."

I'm using the package in one view only:
public let extensionDefaults = UserDefaults(suiteName: "...")!

extension Defaults.Keys {
  public static let isBypass = Key<Bool>("isBypass", default: false, suite: extensionDefaults)
  public static let deviceID = Key<String>("deviceID", default: noneDeviceID, suite: extensionDefaults)
}

struct TabSettingsView: View {
    ...
    @Default(.deviceID) var deviceID: String
    @Default(.isBypass) var isBypass: Bool
    ...
}

Am I using it wrong? Before, in XCode 15, this error did not show up. And I'm using the latest main branch commit.

It's either either a false-positive warning or a bug in SwiftUI. Let's see if Xcode 16.1 fixes it.

Same issue here, any time I even access a Defaults value.

I tried with 8.2.1 and same issue as 9.0.0, when I rollback to 7.1.0 it gone

Can confirm that it 8.0.0 is the culprit. It's not an issue in 7.3.1 but it is in 8.0.0.


Typing ModelStage

State operations ("stages") in an AbstractStagedModel are defined by the property model_spec which is an OrderedDict[str, ModelStage].
For example, here's one entry from SimpleFeedback.model_spec:
https://github.com/mlprt/feedbax/blob/2ce8b1c25b5790dc0a41a5812eeb6d830b3a151f/feedbax/bodies.py#L195-L199
The state arguments in these lambdas should be typed as SimpleFeedbackState, so that the type checker recognizes that state.mechanics is a valid reference: SimpleFeedbackState has a field mechanics: MechanicsState.
Currently, ModelStage is a generic of the type variable StateT = TypeVar('StateT', AbstractState, Array), where all of the state PyTrees like SimpleFeedbackState inherit from AbstractState. Here is a slight simplification:
StateT = TypeVar('StateT', AbstractState, Array)

class ModelStage(eqx.Module, Generic[StateT]):
    callable: Callable[[AbstractStagedModel[StateT]], Callable]
    where_input: Callable[[AbstractTaskInputs, StateT], PyTree]
    where_state: Callable[[StateT], PyTree]
    intervenors: Sequence[AbstractIntervenor] = field(default_factory=tuple)

However, nowhere do we subclass ModelStage and give an argument for this type variable.
Throughout Feedbax, Pyright raises errors for only some of the references found in lambdas of model_spec properties. For example, it raises an error for the callable field of the "mechanics_step" stage given at the start of this issue,
feedbax/bodies.py:196:48 - error: Cannot access member "mechanics" for type "AbstractStagedModel[Unknown]"
    Member "mechanics" is unknown (reportAttributeAccessIssue)

but not for its where_input or where_state fields. Checking the Pylance tooltips, the input arguments are typed as AbstractTaskInputs, but the state fields are Any, which is at odds with the StateT annotation in ModelStage.
I suspect it will be necessary (#24 ) to eliminate AbstractState. Then, the type arguments to final subclasses of AbstractStagedModel will need to be distinct subclasses of equinox.Module. Similarly, the type variable in ModelStage will need to be bound to equinox.Module instead of AbstractState. This doesn't seem problematic as ModelStage will only be used inside an AbstractStagedModel with which it shares a type argument. For example, ModelState[SimpleFeedbackState] will be used in SimpleFeedback[SimpleFeedbackState].
I've tried explicitly writing "mechanics_step": ModelStage[SimpleFeedbackState](...) in the model_spec entries. Writing type arguments in at the time of instantiation is not a syntax I've ever used before, and it doesn't resolve the type checker's errors. I wouldn't expect this to resolve the error raised for the callable field of ModelStage, since the type argument won't specify that self should be of type SimpleFeedback.
How should this be properly typed?

This should be obviated if we replace the staged model approach with a DAG approach (#28), so I'm tagging this issue as "won't fix".


OrientDB Enterprise Agent backup not able to restore on another server

Details
Version: 3.0.42
Enterprise agent 3.0.42
Setup
Since the orientdb enterprise agent became opensource, I tried to build the agent from repository. And generated my own docker image by using following docker file.
FROM orientdb:3.0.42
COPY *.jar /orientdb/plugins/.
RUN addgroup --system --gid 1000 system && adduser orientdb --gid 1000 --uid 1000
RUN chmod ugo+rwx /orientdb && chown -R 1000:1000 /orientdb
USER orientdb
EXPOSE 2424
EXPOSE 2480
CMD ["server.sh"]

I have added a non-root user as orientdb.
Problem statement
When using enterprise server management tool, backup is working and showing on the Server Management dashboard for backup as expected. Restore database is also working as expected.
Now I copy generated backup directory and files to another system and start server, copied backups are not visible anymore on the dashboard of the another system.
On investigating a backups.json file is generated in config directory when configured. I also tried copying backups.json file on another server and still not working.
How do I get my backups visible back to my new server?
This problem is not because of non-root user, as I verified same by executing without docker on windows machine. orientdb zip

Hi,
Are you able to perform a restore using the command line?
Is it correct that you do not see files only in the dashboard?

Hi @laa
I have tried restoring database via commandline but its not working. To restore via command line I use plocal approach to connect to database and perform restore operation because remote connection was not working. Following are my commands:
create database plocal:/orientdb/databases/testdb
restore database /orientdb/backup/<backup-folder>/<backup-file-ibu>

This is deleting the database and then ends up with following error
Database restored in 0.86 seconds
Error: com.orientechnologies.orient.core.exception.OStorageException: Cannot open the storage 'testdb' because it does not exist in path: /orientdb/databases/testdb
        DB name="testdb"

Please let me know if I am doing something wrong here.
To answer your second question, yes the backups are not visible on another server in server management dashboard when backups are copied to other server.
As per my investigation, the logs for these backups are stored in OSystem database under OBackupLog class.
So when I copied this system database to new server, all the backups were visible, given that I also copy the backups.json file which contains the backup UUID to detect the logs associated.
Only combination of both is able to reflect the backups in server management dashboard and also able to restore.
Now OSystem being system database and is recreated automatically everytime. Is there a way out?
Also not sure why, my command line restore is not working.

Hi @laa
Any update on this, please? Is it reproducing on your end?

Resolved.
The restore can be done only on new database. If a new database is created and then applied to restore, it does not work. Restore command itself creates database and restores there.
Refered from docs:
create database remote:localhost/mydb root root plocal graph -restore=/tmp/backup


ARTEMIS-2216 Use a specific executor for pageSyncTimer

Improve paging throughput by using a specific executor for pageSyncTimer
Improving throughput on paging mode is one of our concerns since our cluster uses paging a lot.
We found that pageSyncTimer in PagingStoreImpl shared the same executor with pageCursorProvider from thread pool. In heavy load scenario like hundreds of consumers receiving messages simultaneously, it became difficult for pageSyncTimer to get the executor due to race condition. Therefore page sync was delayed and producers suffered low throughput.
To achieve higher performance we assign a specific executor to pageSyncTimer to avoid racing. And we run a small-scale test on a single modified broker.
Broker: 4C/8G/500G SSD
Producer: 200 threads, non-transactional send
Consumer 200 threads, transactional receive
Message text size: 100-200 bytes randomly
AddressFullPolicy: PAGE
Test result：
| Only Send TPS | Only Receive TPS | Send&Receive TPS
-- | -- | -- | --
Original ver | 38k | 33k | 3k/30k
Modified ver | 38k | 34k | 30k/12.5k
The chart above shows that on modified broker send TPS improves from “poor” to “extremely fast”, while receive TPS drops from “extremely fast” to “not-bad” under heavy load. Considering consumer systems usually have a long processing chain after receiving messages, we don’t need too fast receive TPS. Instead, we want to guarantee send TPS to cope with traffic peak and lower producer’s delay time. Moreover, send and receive TPS in total raises from 33k to about 43k. From all above this trade-off seems beneficial and acceptable.

Comments on this are:
We have shared executor and thread pools as a way to pool these its part artemis design so that these can be controlled, eg the number of threads are configurable. And theres uses cases for why, we dont just add executors. If we add a new pool it should be configurable like others.
Whilst your use case this significantly improves things. As you note this change isnt entirely positive to all users as it does impact use cases where people may care more on the consumer side (i am one one of those). If anything if the broker is saturated its more important consumers for us can dequeue and if anything producers backpressure. As if consumers cant keep up and producer peak continues you can very soon end up situation where paging depth will just grow and grow.
It would be good to get more stats therefore on how as producer rates ramup how consumer rates ramp down. And crticial cross over points. Along with number of remote consumers and producers. All of this could be mute if there is a way to eliminate the negative effect this change has.

@michaelandrepearce As presented in test result adding a specific executor do improve producer rates, and also means page() method in PagingStoreImpl will acquire writelock more often. At the same time QueueImpl::deliver will call isPaging() to check whether queue is paging or not, which acquire readlock and make it a race condition. This may explain why the increase of producer rates is accompanied by the decrease of consumer rates.
According to this we have an internal branch that removes isPaging() method’s readlock in PagingStoreImpl, along with adding pageSyncTimer’s specific executor. After that we performed a same test and get a result of 21k/25k Send&Receive–in total 46k TPS. This version runs smoothly in our use case but we are still exploring potential effects.
The reason why we use transactional receive is to make sure every message indeed delivered and consumed. Say that if we do non-tx receive and consumer system fail in the following processing chain, this message may be delivered but actually “lost”. So we choose to let consumer decide when should they ack and commit, instead of auto-commit acks.

If you could supply more complete stats with ramp ups e.g pef at 1k, 2k, 3k...5k...15k..20k.21k.22k...25k etc etc, over time. Rather than just a summary snapshot.

The receiver could be slower because of more IO. That’s not a bad thing.
I’m on vacation this week. I won’t be able to get to this.  Just saying why the receiver could be slower.

The bit around lock contention on paging check is very interesting where @qihongxu removed it and performance rose, this would suggest consumer perf change isnt related to io but due to over lock contention in code. Def would be good to see if its safe to remove

@qihongxu,
re:
"
According to this we have an internal branch that removes isPaging() method’s readlock in PagingStoreImpl, along with adding pageSyncTimer’s specific executor. After that we performed a same test and get a result of 21k/25k Send&Receive–in total 46k TPS. This version runs smoothly in our use case but we are still exploring potential effects.
"
Agree that as the QueueImpl:checkDepage which is called from deliver method, only schedules another depage asynchronously that the read barrier lock for that is over kill as it doesn't have to be strongly consistent correct.
I think changing the current method could be dangerous, unless we do a fuller analysis on all its uses, but i think we could easily add a new method named something like "isPagingDirtyRead" which can return without the readlock, it will mean the returned value isn't always 100% accurate but in cases such as checkDepage, we don't need it to be, its just a single.
Would you be happy adding this, (will need to be exposed via PageSubscription)? And updating checkDepage to use that one?

@qihongxu,
re:
"
According to this we have an internal branch that removes isPaging() method’s readlock in PagingStoreImpl, along with adding pageSyncTimer’s specific executor. After that we performed a same test and get a result of 21k/25k Send&Receive–in total 46k TPS. This version runs smoothly in our use case but we are still exploring potential effects.
"
Agree that as the QueueImpl:checkDepage which is called from deliver method, only schedules another depage asynchronously that the read barrier lock for that is over kill as it doesn't have to be strongly consistent correct.
I think changing the current method could be dangerous, unless we do a fuller analysis on all its uses, but i think we could easily add a new method named something like "isPagingDirtyRead" which can return without the readlock, it will mean the returned value isn't always 100% accurate but in cases such as checkDepage, we don't need it to be, its just a single, and we can call it where we know its safe to.
Would you be happy adding this, (will need to be exposed via PageSubscription)? And updating checkDepage to use that one?

As you suggested we tried update checkDepage with a new "isPagingDirtyRead" method which can return without the readlock. But after running same tests on this version it seems that checkDepage did not affect receive performance too much. Instead, CursorIterator#hasNext in PageSubscriptionImpl also called isPaging() and had a significant impact on receive performance. According to this we updated hasNext() and forced it to use the new “isPagingDirtyRead”. Below is the result chart.
(P.S the original ver is a control group, not exactly the “original” master-branch build. It has been applied with specific pageSyncTimer executor, and cache durable in PageReference - See PR Artemis-2214.)
| Send&Receive | In Total
-- | -- | --
Ver Original | 30k/12.5k | 32.5k
Ver Modified checkDepage | 31.1k/11.3k | 42.4k
Ver Modified hasNext | 23.8k/23.5k | 47.3k
Detailed TPS chart can be seen in attachment. (Sorry we dont have enough time for running tests. Just got back to work today)
TPS chart.docx

Instead of transaction consumer you could use client acknowledge or even individual acknowledge.

It seems that both client-acknowledge and individual-acknowledge mode will finally use Transaction at server side. Considering these modes have no obvious difference in performance, we choose to use transaction as it’s more reliable and supports rollback.

@qihongxu Just as a confirmation, I have used https://github.com/jvm-profiling-tools/async-profiler on lock events ie time while waiting to enter into a lock and I have found the same exact behaviours explained above:
QueueImpl::checkDepage

QueueImpl::DepageRunner

both are calling PagingStoreImpl::isPaging and are blocking each others

Instead of transaction consumer you could use client acknowledge or even individual acknowledge.

It seems that both client-acknowledge and individual-acknowledge mode will finally use Transaction at server side. Considering these modes have no obvious difference in performance, we choose to use transaction as it’s more reliable and supports rollback.

@qihongxu you're using JMS api not core then?

The version I have shown is just master ie any read lock is there!

The version I have shown is just master ie any read lock is there!

I get that, im more relfecting on this change in PR.

@qihongxu i dont see checkDepage using the dirtyRead in current commit

@michaelandrepearce @qihongxu Just a lil OT but I'm getting these warns on master:
2019-01-03 17:36:44,408 WARN  [org.apache.activemq.artemis.journal] AMQ142007: Can not find record 8,103,136 during compact replay
2019-01-03 17:36:44,408 WARN  [org.apache.activemq.artemis.journal] AMQ142007: Can not find record 8,103,139 during compact replay
2019-01-03 17:36:44,408 WARN  [org.apache.activemq.artemis.journal] AMQ142007: Can not find record 8,103,142 during compact replay
2019-01-03 17:36:44,408 WARN  [org.apache.activemq.artemis.journal] AMQ142007: Can not find record 8,103,145 during compact replay
2019-01-03 17:36:44,408 WARN  [org.apache.activemq.artemis.journal] AMQ142007: Can not find record 8,103,148 during compact replay
2019-01-03 17:36:44,408 WARN  [org.apache.activemq.artemis.journal] AMQ142007: Can not find record 8,103,151 during compact replay
2019-01-03 17:36:44,408 WARN  [org.apache.activemq.artemis.journal] AMQ142007: Can not find record 8,103,154 during compact replay

I have taken a look on the version used for this PR + the change suggested by @michaelandrepearce
on checkDepage and I have found that:
LivePageCacheImpl (in violet) is now a major contention point:

While re isPaging:

and

Are now source of contention

Looking at the CPU graph instead I can see many odd things:

Page writing is double copying the heap ByteBuffer into a native ByteBuffer to write on the page file:

Compaction is stealing lot of cpu and I/O:

during initial development years ago I needed to be careful when to leave paging. this needed some sync points to make sure depage would set it synchronously.
I'm not saying anything against the change (I'm still in vacation mode, I'm back on monday).. just saying what was the original intent when it was written.

@michaelandrepearce

@qihongxu you're using JMS api not core api then?

Yes (sometimes for for compatibility concern)

@qihongxu i dont see checkDepage using the dirtyRead in current commit

Sorry I probably did not make it clear enough in my last comment. As the chart shows we tried checkDepage using the dirtyRead, but found it makes no difference in perf. We even changed isPaging() method in PageSubscriptionImpl from return pageStore.isPaging(); to return pageStore.isPagingDirtyRead();. In this way not only checkDepage, but also other methods that call PageSubscription.isPaging() (as @franz1981 shown in flame charts above) will all use dirtyRead. But still, same perf as before.
From all above we thought it's safer to leave others unchanged, only force CursorIterator:hasNext to use dirtyRead since it affects perf a lot. My latest RP is based on these concerns.

So out the two, the checkDepage is the safer one to use a dirty read, so id expect to see that changed first before anything else.

CursorIterator:hasNext

Im bit concerned with this doing a dirty read, as this isnt something that is trigger an ascyn action, in actual fact the hasNext is purposefully synchronized. (especially as recently some concurrency issues have been found in paging (e.g. howards current pr) im sensitive to us being too hap hazard here.

I don't get why isPaging() in hasNext() needs to be consistent. The paging status can change after isPaging() unless we readlock isPaging() and subsequent operations. Without readlock, a) if isPaging() returns true, but the other thread set it to false, subsequent next() call reads no data and returns null; b) if isPaging() returns false, but the other thread set it to true(a new message coming), deliverAsync would be called later anyway.

if thats the case then the sync method could be removed also, though i think @clebertsuconic has given some further info to why this is like it is.

@franz1981 nice graphs, looks like essentially isPaging in QueueImpl unless everything is dirtyRead then we simply move the problem somewhere else, i think we probably need to make then a more general call if all these are safe to have dirty read or not. (Im starting to feel like

Looking at the CPU graph instead I can see many odd things ie Compaction is stealing lot of cpu and I/O:

Nice find.

@michaelandrepearce @qihongxu Just a lil OT but I'm getting these warns on master:
2019-01-03 17:36:44,408 WARN  [org.apache.activemq.artemis.journal] AMQ142007: Can not find record 8,103,136 during compact replay
2019-01-03 17:36:44,408 WARN  [org.apache.activemq.artemis.journal] AMQ142007: Can not find record 8,103,139 during compact replay
2019-01-03 17:36:44,408 WARN  [org.apache.activemq.artemis.journal] AMQ142007: Can not find record 8,103,142 during compact replay
2019-01-03 17:36:44,408 WARN  [org.apache.activemq.artemis.journal] AMQ142007: Can not find record 8,103,145 during compact replay
2019-01-03 17:36:44,408 WARN  [org.apache.activemq.artemis.journal] AMQ142007: Can not find record 8,103,148 during compact replay
2019-01-03 17:36:44,408 WARN  [org.apache.activemq.artemis.journal] AMQ142007: Can not find record 8,103,151 during compact replay
2019-01-03 17:36:44,408 WARN  [org.apache.activemq.artemis.journal] AMQ142007: Can not find record 8,103,154 during compact replay

Do you get this on master? If not then that is a BIG worry

@michaelandrepearce

Do you get this on master or this PR (im mean is that a typo)?

I've got that on master!

If we return true from the dirty read we can just return it, while if we found the it false we could attempt to enter the read lock and validate that's not paging for real.

Ive literally gone through every case, what occurs is we call isPaging within an if statement, and then do some logic after, as such anyhow any action we do within these if statements anyhow will be based off a stale state.
Im starting to just think we make isPaging not use a read lock  (aka make it dirty), as its only used in queueimpl like mentioned and for queuecontrol (aka the admin gui)

@michaelandrepearce

Do you get this on master or this PR (im mean is that a typo)?

I've got that on master!

ok so i think we not need worry on this for realms of this PR. (probably needs looking into, but doesn;t need to be solved in this pr - imo)

Id agree, im just cautious as we've been hit a few times with concurrency issues that have been a nightmare to find (and quite recently as last month!).
em.....Could you please tell us which issues? We need to verify how it affects our cluster.

@michaelandrepearce
Removed readlock of isPaging().
Also as @franz1981 suggested now only volatile load addressFullMessagePolicy once on each call.
Please review and notify me if any more test needed:)

em.....Could you please tell us which issues? We need to verify how it affects our cluster.

The big issue im relating to, which became a night mare for my organisation, was that under high concurrency (high throughput and low latency broker setup), the buffers can get mixed up, and was causing index out of bounds issues.
Fixes were multiple:
https://github.com/apache/activemq-artemis/commit/024db5bd3c1656d265daf60c9e3a362d53b9088b
https://github.com/apache/activemq-artemis/commit/da7fb89037481fb6343c760010d4553ff28ac87e
Im also aware there have been some other concurrency fixes for smaller issues.

@qihongxu looks good to me.
@franz1981 whats the current PR look like in heat maps?

whats the current PR look like in heat maps?

I'm finishing a thing to fix LivePageCacheImpl too, hopefully today and I will post: impl-wise seems good to me as well :+1:

@michaelandrepearce @qihongxu I have re-implemented the LivePageCache to be completly lock-free: https://github.com/franz1981/activemq-artemis/tree/lock-free-live-page-cache
Fell free to try it into this PR too and I can do a PR to your branch.
I will provide soon some charts of the contention state :+1:

@michaelandrepearce @qihongxu Ok, checked: the latest version of this PR + my branch https://github.com/franz1981/activemq-artemis/tree/lock-free-live-page-cache is never blocked :)

@franz1981 did you send a pr to @qihongxu branch so he can merge it and this pr picks it up?
Be great to see a final stat in @qihongxu test env

@michaelandrepearce Good idea! let me do it now

@michaelandrepearce Done, the PR has been sent, now we can just wait the perf results on it :)
I have improved quite a bit the live page cache behaviour/reliability (especially if OOME), but sadly I see that the most called method getMessage cannot be improved anymore without making the lock-free code a real nightmare.
The original version was O(n) depending which message was queried, because it needs to walk the entire linked list of paged messages.
In my version I have amortized the cost by using an interesting hybrid between an ArrayList and a LinkedList, similar to https://en.wikipedia.org/wiki/Unrolled_linked_list, but (very) optimized for addition.
I'm mentioning this, because is a long time I want to design a single-threaded version of this same data-structure to be used as the main datastructure inside QueueImpl.

@michaelandrepearce @qihongxu Ok, checked: the latest version of this PR + my branch https://github.com/franz1981/activemq-artemis/tree/lock-free-live-page-cache is fully non-blocking :)

Nice work!!

Be great to see a final stat in @qihongxu test env

As soon as I'm back Monday I will try implement same tests on both version(this and franz's).


List it in docker hub?

It doesn't seem to be available on docker hub:
https://hub.docker.com/u/paulbismuth
:o

It is, the authors username has a 0 added on the end on dockerhub. https://hub.docker.com/r/paulbismuth0/nicotine-plus


templates for video sites...

here's a few templates for some other sites if interested...
BitChute: https://www.bitchute.com/feeds/rss/channel/<channel_name>
LBRY/Oddesy: https://lbryfeed.melroy.org/channel/<channel_name>
Steemit: http://www.hiverss.com/@<channel_name>/feed
Vimeo: https://vimeo.com/<channel_name>/videos/rss

Thanks for the info.
If the feeds aren't yet directly exposed by the sites, i'll check if i can add them.

bitchute and vimeo are done.
But for lbryfeed and hiverss i'll have to take a closer look.
Both as far as i can tell seem to be meta services which are not directly provided by the sites themself.
I am not sure if adding/using them is a very good idea.
But i'll think about it.

Some feeds are now added, there might be some missing.
I'll be waiting for the bug reports.
Later.

the Steemit one was a poor suggestion on my part since the feed is built by a 3rd party (hiverss.com) - vimeo and bitchute were the more important ones ... thanks!


Durin icon not showing up correctly on android

Installed from google play store, tested on Pixel 6 and Fairphone 4

@meandavejustice Is this still an issue? Maybe something @drewdelano can look into

No this was fixed


Updated MutableMapping import for Python 3.10 compatibility.

Changed how MutableMapping is imported depending on the Python version.
Tested with Python 3.7 and Python 3.10 to ensure it works.
refs: #13

Hi thanks for the PR!
I haven't looked at this project in many years, so I'm happy to merge. Would you like to be added as a contributor to this project?

@mrkkollo, a new version of the project with your changes is now on PyPi


How can I adjust the token limit for input?

What is the maximum input the model can handle for transcription? GPT-4 has a 128k token limit, but it is not clear how much of this is used for generating the response.

Transcription and response generation are different functionality independent of each other.

What is the maximum input the model can handle for transcription?
Transcription is done on audio files. While we can provide a prompt for transcription as well, this isn't something transcribe does.

As you correctly noted, the input token limit for response generation depends on the model.

@mang0sw33t I am interested in token limit for response generation. How does transcribe know token limit of a particular model I chose? If I go with Gemini 1.5 pro with token limit 1M will response generation take into account that the model I use has 1M token limit?

For LLM responses, Transcribe does not do any automatic limits on input tokens based on the model.
MAX_TRANSCRIPTION_PHRASES_FOR_LLM is how many previous conversations as visible in the UI, are sent to LLM. You are correct, setting it to a higher number will result in more tokens sent to LLM.
Please consider a PR with making this parameter configurable so other users can change it easily in parameters.yaml file or override.yaml file


Mongodb issues econnrefused with a clean repo

Issue description
I'm following the associated tutorial and downloaded the seed branch. I ran npm install. I tried node app.js and received this error:

Listening on port 3000
mypath\node_modules\mongodb\lib\server.js:242
process.nextTick(function() { throw err; })
^
Error: connect ECONNREFUSED <IP_ADDRESS>:27017
at Object._errnoException (util.js:1022:11)
at _exceptionWithHostPort (util.js:1044:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1198:14)
PS mypath>

It doesn't work on other random ports either, like 1885. Is there a firewall setting or something? Also, I read a post suggesting that mongoose is a server. If I'm expected to download and run it as a server, please, please cover these things in the initial tutorial; there are too many node packages to assume familiarity with any particular set.
Environment

Windows 10, 64-bit

HI @JoshuaLamusga
Did you get on the wrong repository maybe? Because there is no MongoDB in the MarkdownViewer++ ^^
Just guessing: On the top its listening to port 3000, below connection is being established to 27017. Maybe somethings mismatched.
Cheers


Feature Request: Add support for pounce.nvim

pounce.nvim is similar to lightspeed/hop, but it uses fuzzy matching. The highlight groups currently are PounceMatch, PounceGap and PounceAccept. There's also the currently experimental PounceAcceptBest.
Would be really cool if the theme supported this out of the box! :)

Feel free to create a PR


Change findAndModify  by findOneAndUpdate

(node:4252) DeprecationWarning: collection.ensureIndex is deprecated. Use createIndexes instead.
warning.js:18
(node:4252) DeprecationWarning: collection.findAndModify is deprecated. Use findOneAndUpdate, findOneAndReplace or findOneAndDelete instead.
warning.js:18
(node:4252) UnhandledPromiseRejectionWarning: #
warning.js:18
(node:4252) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
warning.js:18
(node:4252) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
warning.js:18

Fixed by @daton89 with commit 91763c3e9bd7536e869c5d868269d884f46603a2


MCUboot support for SAM E70 Xplained

Summary
Initial support of MCUboot for SAMv7 based boards with SAM E70 Xplained as an example
Impact
SAMv7 board get MCUboot capabilities
Testing

MCUboot loader works on custom SAME70 based board

@pkarashchenko there is a minor style warning:
https://github.com/apache/incubator-nuttx/runs/4312524794?check_suite_focus=true
Please fix it, thanks.

@pkarashchenko there is a minor style warning: https://github.com/apache/incubator-nuttx/runs/4312524794?check_suite_focus=true Please fix it, thanks.

I'm not sure what is wrong here and how to fix it. ((void (*)(void))vt.reset)(); seems to be fine to me. Could you please advice how code should look like?

@pkarashchenko there is a minor style warning: https://github.com/apache/incubator-nuttx/runs/4312524794?check_suite_focus=true Please fix it, thanks.

Done

Currently build fails because of -Werror is used for PreCI builds. The fix is submitted to MCUboot and waiting to be merged: https://github.com/mcu-tools/mcuboot/pull/1222


Fix default number of genes for marker_genes_overlap

This should fix #1411 that became an issue when DE results for all genes are stored in sc.tl.rank_genes_groups instead of only top 100 by default.

Rank genes groups results are still ordered though, right?

Yes, they are.

Then this should be ready to merge

@LuckyMD Could you also add this fix to the release notes?
https://github.com/theislab/scanpy/blob/master/docs/release-latest.rst

done

Thanks.


macOS build

readme.md:

@stonedreamforest That's why I created an issue

I've successfully built bin2llvmir & llvmir2hll on macOS:
$ clang --version
Apple LLVM version 9.0.0 (clang-9<IP_ADDRESS>)
Target: x86_64-apple-darwin17.2.0

I had to install a few packages, use homebrew's bison, fix a few compilation issues & add build script support. There are so many submodules, I don't know where to start filing pull requests, so attached is a recursive diff of my source tree.
macOS.patch
Reproduced below:
diff --git a/deps/openssl/CMakeLists.txt b/deps/openssl/CMakeLists.txt
index afa11d3..ec62f93 100644
--- a/deps/openssl/CMakeLists.txt
+++ b/deps/openssl/CMakeLists.txt
@@ -15,6 +15,8 @@ endif()

 if(WIN32)
 	set(OS_WINDOWS 1)
+elseif(APPLE)
+    set(OS_MACOS 1)
 elseif(UNIX)
 	set(OS_LINUX 1)
 else()
@@ -27,7 +29,7 @@ if(OS_WINDOWS)
 	else()
 		message(ERROR "Unsupported compiler. Supported: MSVC.")
 	endif()
-else(OS_LINUX)
+elseif(OS_LINUX)
 	if("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")
 		set(COMPILER_GCC 1)
 	elseif("${CMAKE_CXX_COMPILER_ID}" STREQUAL "Clang")
@@ -46,6 +48,13 @@ if(OS_WINDOWS)
 			set(OPENSSL_CONFIGURE_ARCH "VC-WIN64A")
 		endif()
 	endif()
+elseif(OS_MACOS)
+    set(OPENSSL_MAKE make -j${CPUS})
+	if(ARCH_32BIT)
+        set(OPENSSL_CONFIGURE_ARCH "darwin-i386-cc")
+	elseif(ARCH_64BIT)
+        set(OPENSSL_CONFIGURE_ARCH "darwin64-x86_64-cc")
+	endif()
 elseif(OS_LINUX)
     set(OPENSSL_MAKE make -j${CPUS})
 	if(ARCH_32BIT)
diff --git a/include/fileformat/file_format/intel_hex/intel_hex_parser/intel_hex_parser.h b/include/fileformat/file_format/intel_hex/intel_hex_parser/intel_hex_parser.h
index 1db74fe..2a52c7a 100644
--- a/include/fileformat/file_format/intel_hex/intel_hex_parser/intel_hex_parser.h
+++ b/include/fileformat/file_format/intel_hex/intel_hex_parser/intel_hex_parser.h
@@ -31,7 +31,7 @@ class IntelHexSection

 		/// @name Operators
 		/// @{
-		bool operator<(IntelHexSection const &a);
+		bool operator<(IntelHexSection const &a) const;
 		/// @}
 };

diff --git a/src/ar-extractor/archive_wrapper.cpp b/src/ar-extractor/archive_wrapper.cpp
index 3a654eb..86e2b10 100644
--- a/src/ar-extractor/archive_wrapper.cpp
+++ b/src/ar-extractor/archive_wrapper.cpp
@@ -227,7 +227,7 @@ bool ArchiveWrapper::getJsonList(
 			Value object(kObjectType);
 			object.AddMember("name", Value(outName.c_str(), allocator).Move(), allocator);
 			if (numbers) {
-				object.AddMember("index", counter++, allocator);
+				object.AddMember("index", (uint64_t)counter++, allocator);
 			}
 			objects.PushBack(object, allocator);
 		}
diff --git a/src/fileformat/file_format/intel_hex/intel_hex_parser/intel_hex_parser.cpp b/src/fileformat/file_format/intel_hex/intel_hex_parser/intel_hex_parser.cpp
index 5c73935..12704da 100644
--- a/src/fileformat/file_format/intel_hex/intel_hex_parser/intel_hex_parser.cpp
+++ b/src/fileformat/file_format/intel_hex/intel_hex_parser/intel_hex_parser.cpp
@@ -32,7 +32,7 @@ IntelHexSection::~IntelHexSection()
 /**
  * operator <
  */
-bool IntelHexSection::operator<(const IntelHexSection &a)
+bool IntelHexSection::operator<(const IntelHexSection &a) const
 {
 	return address < a.address;
 }
diff --git a/src/unpackertool/CMakeLists.txt b/src/unpackertool/CMakeLists.txt
index 94ce8d5..0c7ba44 100644
--- a/src/unpackertool/CMakeLists.txt
+++ b/src/unpackertool/CMakeLists.txt
@@ -16,7 +16,9 @@ if(NOT TARGET unpackertool)
 	# 3. The libraries that are used in plugins under Windows are linked to plugins themselves in plugins/*/CMakeLists.txt
 	# 4. The libraries that are used both by core and plugins need to be linked as whole-archive under Linux, and under Windows like in 1 and 3 simultaneously
 	target_link_libraries(unpackertool cpdetect)
-	if(UNIX)
+    if(APPLE)
+		target_link_libraries(unpackertool -rdynamic -Wl,-force_load tl-cpputils unpacker loader pelib dl)
+    elseif(UNIX)
 		target_link_libraries(unpackertool -rdynamic -Wl,-whole-archive tl-cpputils unpacker loader pelib -Wl,-no-whole-archive dl)
 	elseif(MSVC)
 		target_link_libraries(unpackertool shlwapi)
diff --git a/src/unpackertool/plugins/mpress/CMakeLists.txt b/src/unpackertool/plugins/mpress/CMakeLists.txt
index c90c342..950faa6 100644
--- a/src/unpackertool/plugins/mpress/CMakeLists.txt
+++ b/src/unpackertool/plugins/mpress/CMakeLists.txt
@@ -9,6 +9,8 @@ target_include_directories(mpress PUBLIC ${PROJECT_SOURCE_DIR}/src/)
 # Plugin related libraries are linked to the plugin on Windows
 if(MSVC)
 	target_link_libraries(mpress unpacker)
+elseif(APPLE)
+	target_link_libraries(mpress unpacker)
 endif()
 install(TARGETS mpress
 	LIBRARY DESTINATION bin/unpacker-plugins
diff --git a/src/unpackertool/plugins/upx/CMakeLists.txt b/src/unpackertool/plugins/upx/CMakeLists.txt
index f4f34a9..94c5c78 100644
--- a/src/unpackertool/plugins/upx/CMakeLists.txt
+++ b/src/unpackertool/plugins/upx/CMakeLists.txt
@@ -21,6 +21,8 @@ target_include_directories(upx PUBLIC ${PROJECT_SOURCE_DIR}/src/)
 # Plugin related libraries are linked to the plugin on Windows
 if(MSVC)
 	target_link_libraries(upx tl-cpputils unpacker)
+elseif(APPLE)
+	target_link_libraries(upx tl-cpputils unpacker)
 endif()
 install(TARGETS upx
 	LIBRARY DESTINATION bin/unpacker-plugins
diff --git a/deps/fileformat b/deps/fileformat
--- a/deps/fileformat
+++ b/deps/fileformat
@@ -1 +1 @@
-Subproject commit 737baad9cd2bc1238f6d5229707ecdf1606c3405
+Subproject commit 737baad9cd2bc1238f6d5229707ecdf1606c3405-dirty
diff --git a/deps/openssl/CMakeLists.txt b/deps/openssl/CMakeLists.txt
index afa11d3..ec62f93 100644
--- a/deps/openssl/CMakeLists.txt
+++ b/deps/openssl/CMakeLists.txt
@@ -15,6 +15,8 @@ endif()

 if(WIN32)
 	set(OS_WINDOWS 1)
+elseif(APPLE)
+    set(OS_MACOS 1)
 elseif(UNIX)
 	set(OS_LINUX 1)
 else()
@@ -27,7 +29,7 @@ if(OS_WINDOWS)
 	else()
 		message(ERROR "Unsupported compiler. Supported: MSVC.")
 	endif()
-else(OS_LINUX)
+elseif(OS_LINUX)
 	if("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")
 		set(COMPILER_GCC 1)
 	elseif("${CMAKE_CXX_COMPILER_ID}" STREQUAL "Clang")
@@ -46,6 +48,13 @@ if(OS_WINDOWS)
 			set(OPENSSL_CONFIGURE_ARCH "VC-WIN64A")
 		endif()
 	endif()
+elseif(OS_MACOS)
+    set(OPENSSL_MAKE make -j${CPUS})
+	if(ARCH_32BIT)
+        set(OPENSSL_CONFIGURE_ARCH "darwin-i386-cc")
+	elseif(ARCH_64BIT)
+        set(OPENSSL_CONFIGURE_ARCH "darwin64-x86_64-cc")
+	endif()
 elseif(OS_LINUX)
     set(OPENSSL_MAKE make -j${CPUS})
 	if(ARCH_32BIT)

Now I just need to figure out how to use it...

@mabrowning but why?.. there's #17, everything was done already :}

Pulled down the code a week ago when first announced, started porting to macOS, came back a week later and didn't check if anyone else had done the work. Finished up and posted my results..... oops!

I have merged all PRs related to macOS build in all our repositories. Let me know if I missed anything. Also, I added disclaimer that we do not officially support macOS, because we have no way to check whether macOS build is passing or not. We will change that as soon as we will get some macOS machine. Thank you all for assistance.

At least one fix seems to be missing:
Scanning dependencies of target unpackertool
[ 60%] Building CXX object deps/fileformat/src/unpackertool/CMakeFiles/unpackertool.dir/arg_handler.cpp.o
[ 60%] Building CXX object deps/fileformat/src/unpackertool/CMakeFiles/unpackertool.dir/unpacker.cpp.o
[ 60%] Building CXX object deps/fileformat/src/unpackertool/CMakeFiles/unpackertool.dir/plugin_mgr.cpp.o
[ 60%] Linking CXX executable unpacker
ld: unknown option: -force-load
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [deps/fileformat/src/unpackertool/unpacker] Error 1
make[1]: *** [deps/fileformat/src/unpackertool/CMakeFiles/unpackertool.dir/all] Error 2
make: *** [all] Error 2

Try now (master at 583292145ba28ccfbd612a0afd13edd1fe5720f1). I noticed that the linker flag is actually called force_load, not force-load.

Yes, it now compiles without any local changes. Great job!

Just out of curiousity - how did you avoid that the original Apple flexversion got picked up? My build was failing until I renamed the original Apple files.

I just pointed my $PATH at them in the she'll I was building in. brew install flex mentions that option.
Alternatively, brew link flex will symlink into /usr/local/bin to
override everywhere.
On Dec 31, 2017 8:34 AM, "newkit"<EMAIL_ADDRESS>wrote:

Just out of curiousity - how did you avoid that the original Apple
flexversion got picked up? My build was failing until I renamed the
original Apple files.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/avast-tl/retdec/issues/7#issuecomment-354606960, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAMqRMIog3YuHhPGy_LMa_1WN4lQsF8Sks5tF5togaJpZM4Q_4LJ
.

Okay, thanks for clarification. Method 1 was what I was doing but apperently that was not enough or maybe the order (appending at the end / inserting at beginnung of $PATH) is key.

@newkit Reading the README might be useful.
# Apple ships old Flex & Bison, so Homebrew versions should be used.
export CMAKE_INCLUDE_PATH="/usr/local/opt/flex/include"
export CMAKE_LIBRARY_PATH="/usr/local/opt/flex/lib;/usr/local/opt/bison/lib"
export PATH="/usr/local/opt/flex/bin:/usr/local/opt/bison/bin:$PATH"

I know this sounds like I am unable to do proper RTFM'ing. But I just double checked. I had set everything up as  described in the README and ran into the issue due to the old flex version picked up by yaramod.


basic_thread_border_router build fails with ethernet support (TZ-678)

Using documentation from https://docs.espressif.com/projects/esp-thread-br/en/latest/dev-guide/build_and_run.html basic_thread_border_router build fails when disabling wifi and enabling ethernet support with error:

fatal error: esp_eth.h: No such file or directory
17 | #include "esp_eth.h"
|          ^~~~~~~~~~~
compilation terminated.
Compilation failed because protocol_examples_common.h (in "protocol_examples_common" component) includes esp_eth.h, provided by esp_eth component(s).
However, esp_eth component(s) is not in the requirements list of "protocol_examples_common".
To fix this, add esp_eth to REQUIRES list of idf_component_register call in /code/thread-border-router/esp-idf/examples/common_components/protocol_examples_common/CMakeLists.txt.

couldn't get it working by adding esp_eth to the CMakeLists, any guidance how to build it?

Hi @purha , could you please share which idf commit and esp-thread-br commit did you use? Did you try the idf branch v5.1.2? I just compile the esp-thread-br[commit c7f010397] with the idf v5.1.2[commit 482a8fb2d7], disable the config EXAMPLE_CONNECT_WIFI, enable the config EXAMPLE_CONNECT_ETHERNET, no errors come out.

For esp-idf, I use v5.1.2 / 482a8fb2d7, and for esp-thread-br I'm using latest from main, tried commit c70103 but the same issue remains (ran clean beforehand)
here's the relevant lines from sdkconfig:

CONFIG_EXAMPLE_CONNECT_WIFI is not set
CONFIG_EXAMPLE_CONNECT_ETHERNET=y
CONFIG_EXAMPLE_ETHERNET_EMAC_TASK_STACK_SIZE=2048
CONFIG_EXAMPLE_USE_SPI_ETHERNET=y
CONFIG_EXAMPLE_USE_DM9051 is not set
CONFIG_EXAMPLE_USE_W5500=y
CONFIG_EXAMPLE_USE_OPENETH is not set

I'm trying to build this on mac os sonoma, apple m1
here's the error:

[ 98%] Built target __idf_wifi_provisioning
[ 98%] Built target __idf_espressif__esp_ot_cli_extension
[100%] Building C object esp-idf/thread_border_router/CMakeFiles/__idf_thread_border_router.dir/src/border_router_launch.c.obj
[100%] Linking C static library libesp_ot_br_server.a
[100%] Built target __idf_esp_ot_br_server
In file included from /code/thread-border-router/esp-thread-br/examples/common/thread_border_router/src/border_router_launch.c:42:
/code/thread-border-router/esp-idf/examples/common_components/protocol_examples_common/include/protocol_examples_common.h:17:10: fatal error: esp_eth.h: No such file or directory
17 | #include "esp_eth.h"
|          ^~~~~~~~~~~
compilation terminated.
make[3]: *** [esp-idf/thread_border_router/CMakeFiles/__idf_thread_border_router.dir/src/border_router_launch.c.obj] Error 1
make[2]: *** [esp-idf/thread_border_router/CMakeFiles/__idf_thread_border_router.dir/all] Error 2
make[1]: *** [CMakeFiles/flash.dir/rule] Error 2
make: *** [flash] Error 2
Compilation failed because protocol_examples_common.h (in "protocol_examples_common" component) includes esp_eth.h, provided by esp_eth component(s).
However, esp_eth component(s) is in the private requirements list of "protocol_examples_common".
To fix this, move esp_eth from PRIV_REQUIRES into REQUIRES list of idf_component_register call in /code/thread-border-router/esp-idf/examples/common_components/protocol_examples_common/CMakeLists.txt.
make failed with exit code 2, output of the command is in the /code/thread-border-router/esp-thread-br/examples/basic_thread_border_router/build/log/idf_py_stderr_output_8843 and /code/thread-border-router/esp-thread-br/examples/basic_thread_border_router/build/log/idf_py_stdout_output_8843

here's the stderr:

Compiler supported targets: xtensa-esp32s3-elf
In file included from /code/thread-border-router/esp-thread-br/examples/common/thread_border_router/src/border_router_launch.c:42:
/code/thread-border-router/esp-idf/examples/common_components/protocol_examples_common/include/protocol_examples_common.h:17:10: fatal error: esp_eth.h: No such file or directory
17 | #include "esp_eth.h"
|          ^~~~~~~~~~~
compilation terminated.
make[3]: *** [esp-idf/thread_border_router/CMakeFiles/__idf_thread_border_router.dir/src/border_router_launch.c.obj] Error 1
make[2]: *** [esp-idf/thread_border_router/CMakeFiles/__idf_thread_border_router.dir/all] Error 2
make[1]: *** [CMakeFiles/flash.dir/rule] Error 2
make: *** [flash] Error 2

I am seeing this same error when building on Linux using esp-idf 5.1.2
When I build with commit ec65dc8a17a6a4a0006871d3c5a7d5ba43801486 then the build is successful
When I build with commit c7f01039780fb2f72cfcd8f015ce52e922469ab0 then the build fails, with the error as described.
The configuration changes I have made are select OPENTHREAD_RADIO_SPINEL_UART, enable OPENTHREAD_BR_AUTO_START and OPENTHREAD_BR_START_WEB, disable EXAMPLE_CONNECT_WIFI, enable EXAMPLE_CONNECT_ETHERNET (and set ethernet SPI device configuration)

I am seeing this same error when building on Linux using esp-idf 5.1.2
When I build with commit ec65dc8 then the build is successful When I build with commit c7f0103 then the build fails, with the error as described.
The configuration changes I have made are select OPENTHREAD_RADIO_SPINEL_UART, enable OPENTHREAD_BR_AUTO_START and OPENTHREAD_BR_START_WEB, disable EXAMPLE_CONNECT_WIFI, enable EXAMPLE_CONNECT_ETHERNET (and set ethernet SPI device configuration according to documentation)

Hi, did you try the esp-thread-br v1.0 tags? In our docs, we recommend to use esp-thread-br tags v1.0 with esp-idf tags v5.1.2.

The v1.0 tag works with esp-idf v5.1.2, yes.
Should I use a different version of esp-idf if i want to build the main branch of esp-thread-br? Which version, 5.2.1?

Hi, @kepstin I forgot to enable OPENTHREAD_BR_AUTO_START, with enabling this, I reproduce this issue. Will fix it asap.

The compile issue has been fixed by https://github.com/espressif/esp-thread-br/commit/6d1b5662dca0a0230095a3314ec50e2ff85ca828


Annotation Filters

If an annotation is added for multiple slides, filtering by type shows it for the first slide only.
e.g. an external resource annotation spans more than one slide, filtering by annotation type only works for the first slide where this annotation occurs, not the other slides

this is how it is implemented initially