Optionally use Java VM from "JAVA_BIN" environment variable on *nix systems

Hi.
This patch enables the possibility to run CrateDB on other operating systems than the ones it has been built for with respect to the Java VM bundling. Specifically, I am aiming for easy invocation on FreeBSD.
With kind regards,
Andreas.

I'm not sure if it's a good idea to allow users to use a custom JDK instead of the bundled one. That opens up the door for more JDK related issues. We only do QA with the version we bundle.

Hi Mathias,
I completely understand your reasoning and I share the same thoughts about potential pitfalls on this matter. Likewise, I well understand the burden of maintenance in this spot.
Still I wanted to have this patch around for people from the community needing it, so I decided to submit it. Maybe we will come back to this at some point in the future.
With kind regards,
Andreas.

All right ;]. Reopening and tagging @yurivict, because he seems to be active within the FreeBSD community, see #9027. Maybe he has general interest or wants to share some feedback.

Dear Yuri,
thanks for your response. In order to shed more light onto this, the topic we are discussing here is related to runtime support.
Instead of the topic of building CrateDB, as you shared on behalf of #9027, this is related to add support for running CrateDB from the tarball distribution (originally built for Linux only) on FreeBSD. The only OS-specific thing inside the tarball distribution is the JVM, which is now bundled into the release archive.
As the other artefacts are "just" Java bytecode, it can run on any compatible JVM distribution. My idea was to bring back support to the invocation wrapper script ./bin/crate to be able to run on any JVM as designated through the JAVA_BIN environment variable.
With kind regards,
Andreas.

[...] this is related to add support for running CrateDB [...]

Sorry for my misunderstanding.
IMO it is fine to have JAVA_BIN to define what Java is used at run-time.

I'm not aware of any OS setting a JAVA_BIN environment variable. Isn't the intention more to lookup the java executable and try to extract the JAVA_HOME path from it?

Why not just use JAVA_HOME instead?

We default to the bundled JDK, so we'd need another flag like FORCE_SYSTEMJDK - and based on the presence of that we could use JAVA_HOME instead of the bundled JDK.
But I'm still not convinced if it is a good idea to allow that.

Hi again,
after some more discussions, a reasonable way to implement this would be to introduce a dedicated CRATE_USE_SYSTEMJDK environment variable to augment the runtime behavior, which acts similar like the useSystemJdk Gradle option augmenting the build-time behavior [1].
In order to reduce clutter on the issue tracker, we are closing this and might come back to the topic on behalf of a different patch. At the same time, we are open to hear from the community about any needs of this feature, like, the need to run CrateDB on FreeBSD and friends or different scenarios.
With kind regards,
Andreas.
[1] https://github.com/crate/crate/blob/2a300d4fb0776accf1e17e3c35f3b032ca2cc2cf/gradle/jdkRuntime.gradle#L28-L57


Rtcp rewrite

Changes:

Fixed error in cpptest-lite
added NTP timestamps for RTCP
RT(C)P now uses network byte-order (WIP)
improved some tests
greatly improved RTCP-packages
moved RTCP to own thread and port
program now sends RTCP packages in fixed intervals

@Kaayy-J, @J0n8s someone please review and merge


Update packages

Contact Details
<EMAIL_ADDRESS>Explain the proposed feature
Update packages. The main update is a major Next version. This will eliminate several vulnerabilities.
Explain the possible use cases for the feature
Fewer vulnerabilities.
Any other details you would like to provide?
No response

@dlford created this issue to update packages.

Getting an error after updating to Next.js 13, apparently I'm not the only one:
Error: Hydration failed because the initial UI does not match what was rendered on the server.
I disabled my browser extensions to make sure it wasn't related to anything they were doing. The site text is also randomly getting translated to different languages, and the language dropdown is on the left side of the page, so I think updating may have broken several things. I am going to try to solve all the issues:

spontaneous translation
language menu placement
Hydration failed

@wesleyhitson you might find some good information here on the upgrade - https://nextjs.org/docs/pages/building-your-application/upgrading/version-13

@dlford I looked through that and that helped, thank you. I was able to fix the hydration error, and the language menu seems to be in the correct location.
Maybe you can help me understand more about the i18n setup. When running in localhost, when I click on a different page, in the right menu, the new page loads, but in the first language available for translation. When I refresh the page, it loads with the correct language. The translated pages seem to still be under the "normal" URLs (as opposed to //). The right menu also has a mix of languages in it.
Additionally, I can select different menus from the dropdown, but the language translated to appears to be random. It almost seems like i18n is conflicted with itself somehow.

@wesleyhitson that is strange, is your locale translated or missing? The default locale is en-US, so if your locale is not translated it should show it in en-US as a fallback.

I checked that and that was indeed my locale, so I'm not sure. Could be a weird bug with the Nextjs 13 upgrade. Funny enough I saw Nextjs 14 was just released. 13's big thing seems to be getting everyone to switch the application structure to use an app/ folder, and I'm not so sure I want to restructure the whole project just for that haha. Let me see if I'm still getting the translation behavior if I don't do any updates at all. I'll get back to you @dlford


GH-444: Adds a disabled attribute to EmbeddedKafka

Allows classes to disable the instantiation of KafkaEmbedded on a case-by-case basis.
Resolves https://github.com/spring-projects/spring-kafka/issues/444

I think would be better to just remove @Inherited from the @EmbeddedKafka and let end-user to decide when and how to specify it in his/her hierarchy.
But good catch anyway!
Sad that we didn't do that before GA 😄

Yeah, it seems a bit awkward either way.  It's nice to be able to inherit it, but awkward to disable it.    I'm open to whatever makes the most sense for the project!

Sorry, I would deny this disable approach and would leave it as @Inherited.
It's really a bit awkward to enable feature to disable it :shrug:.
That's definitely problem of the target hierarchy to inherit Kafka aware suit, but don't use it.
Smells like a flaw of the target design...

Understandable!


Improve installation guide suggestion

Details
As a windows user, when following the installation guide for Other OS: source venv/bin/activate, I get the error output for no source command and no such directory.
Expected Result

Virtual environment is activated.
The active file is in \venv\bin\ folder

Actual Result

Error occurs.
The activate file is in \venv\Scripts\ folder

Solution
After reading the docs of virtualenv userguide, I find that the activate file for virtualenv is in \venv\Scripts\ folder for windows only, but \venv\bin\ folder for other operating systems. Also windows prohibits the scripts execution.
Following steps helps:

Execute the powershell as an administrator.
Relax system’s script execution policy to AllSigned by command Set-ExecutionPolicy AllSigned
Relax the system execution policy to allow running of local scripts without verifying the code signature using the command:Set-ExecutionPolicy RemoteSigned
Activate the virtualenv with the command: .\venv\Scripts\activate

More, if using cmd, you can just skip the script execution policy configuration and activate the virtualenv with the command: .\venv\Scripts\activate
Hope
I hope the starting guide for installation could be more consummate and if you want, I could make a pull request for updating the doc.

Actually, RTD doesn't run on Windows, we use symbolic links and other unixlike commands. I think there is a way of using something like that on Windows, but I don't think it will work with the default cmd/powershell

Ops, I see that. It works rendering the index page, but could not work for other fucntions.

We still can have some notes for windows users, like using the linux subsystem could work, but I don't have experience with that.

Actually, windows provides WSL(windows subsystem for linux) to match the requirement of running native Linux command-line tools directly on Windows. I'm trying to find whether it could works if RTD runs in the WSL and is retrieved in windows browser. I would post the result later.

Unfortunately, I've tried the WSL, but it gets the same result while RTD serving in windows.
It always returns HTTP connection error with status code 500. The raising error shows new connection could not be established(I search). I could not solve the problem.
Below are the error images:

@xwy27 you need elasticsearch installed and running on your local machine to use the search functionality.

@saadmk11 Yes, I found that the elasticsearch is not activate. But the error output even occur when I try to login with Github. It still has a HTTP connection error.

@xwy27 you need to setup allauth for that. You can use ngrok for that as you need a live server for it. I would suggest importing projects manually for development or use the test data.


Odds calculator

A calculator that calculates odds of winning and odds of losing.
I would like to add this calculator

Go ahead
Deadline 25 May,2021


Update organization URLs for sbg-driver

Hello ROS-team,
In collaboration with @ThomasLeMezo, and the organization @ENSTABretagneRobotics, the sbg-driver for ROS has changed organization (has been transfered), to be developped and maintained by the @SBG-Systems organization.
For that, the distributions.yaml files have been updated with the new URLs.
I hope all changes are fine,
Thanks in advance,
Best regards,
Erwan

Thanks for updating the urls.


pre-commit: test PR119527

PR Link: https://github.com/llvm/llvm-project/pull/119527

The corpus is out of date.


[VOTE]next version what to do

Hi, everybody. In the email about what we need to do in the next version, we collected the following work. We hope that everyone can discuss the priority of related issues and the specific implementation of the plan. Each job is accompanied by related proposers and advisers. If you are interested, you can openly communicate with them. (You can communicate via email for a certain sub-item)
[0]1:Api Server communicate for Master instand of scan command table( @qiaozhanwei  )
[2]2:Alert module support service for other service invoke( @qiaozhanwei  @gaojun2048  )
[2]3:SQL task support multiple sqls and parameter passing optimization(we should add a new task type, such as SqlScript @gabrywu  @qiaozhanwei  @yangyichao-mango  )[need discuss]
[0]4:Support passing parameters between tasks ( @qiaozhanwei  )
[2]5:Sort out and simplify the license process
[2]6:Focus on fixing bugs base on the branch some new features can be released in 1.4.0_release base on branch 'dev'.(@gabrywu )
[0]7:Expect dolphinScheduler to provide SSO to facilitate seamless integration with existing platforms.( @yangyichao-mango  )
[3]8:SPI requirements are collected and developed. We need to have a discussion to find out the modules that currently need SPI most( @gaojun2048  @lenboo @CalvinKirs )
[0]9:Worker Poll slot(@yangyichao-mango )
[0]10:state machine processor of worker and master optimize( @yangyichao-mango )
[2]11:How to do project migration.Including workflow definitions, resource files, and data sources.The development environment and the production environment are two sets of environments.(@BoYiZhang)
12:Adding a batch stop on the process instance page(@xingchun-chen )
13:Many new functions have been added in the dev branch and some issues have been solved. It is recommended to merge them into the next version(@all)
You can list the digital subscripts of the functions you expect, and then reply directly to the email. We will determine the main work content for the next issue based on your opinions. We look forward to your suggestions. thanks
Voting email address：https://lists.apache.org/thread.html/r931baa57099e6ddd639d1b8c7a33c3e3b8d03352fb0de82b45a7dfb3%40<dev.dolphinscheduler.apache.org>
Best  wishes！
CalvinKirs

2:Alert module support service for other service invoke( @qiaozhanwei @gaojun2048 )
3:SQL task support multiple sqls and parameter passing optimization(we should add a new task type, such as SqlScript @gabrywu @qiaozhanwei @yangyichao-mango )[need discuss]
6:Focus on fixing bugs base on the branch some new features can be released in 1.4.0_release base on branch 'dev'.(@gabrywu )
5:Sort out and simplify the license process(@CalvinKirs )
8:SPI requirements are collected and developed. We need to have a discussion to find out the modules that currently need SPI most( @gaojun2048 @lenboo @CalvinKirs )
11:How to do project migration.Including workflow definitions, resource files, and data sources.The development environment and the production environment are two sets of environments.(@BoYiZhang)

Is everyone doing the issue : Alert module support service for other service invoke  ? I don`t found the issue about it.


Type of expression is ambiguous without more context

PromiseKit version: 6
Xcode version: 9.2
CocoaPods version: 1.4.0

So this is the code.
try service.fetchAll().then({ (result) in
      self.projects = result.projects
})

error: Type of expression is ambiguous without more context
happens on line: swift self.projects = result.projects
function fetchAll() return type is a Promise<ProjectsResponse> which contains the projects variable
self.projects and result.projects are same data type, and even the compiler auto completes the .projects value for me, correct with data type and code highlighting.
When I come to build time, I receive this error, which is really confusing.

See our release notes and migration guide, you need to use done, then requires you to return a promise.
Next time something is confusing, check the function signature of the thing you are using (then here), it would make it clear what is expected of you when Swift is incapable of telling you.


libffi: fix musl tests

Tests do not compile on musl, also alpine seems not to run tests.

Motivation for this change
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

cc @dtzWill


disable min/max page values per column

We're writing html files to parquet. Right now, page statistics are being written with the "min" and "max" values, but that isn't useful for a block of multiple megabytes of html, but it does bloat the file size.
There's a similar ticket for Java.
https://issues.apache.org/jira/browse/PARQUET-1911
My thought would be to add a new struct tag option, nostats, nobounds, skipbounds or something like that, and skip the min/max calculations. Would you be opposed to supporting that?

I added a quick PR to see if I'm approaching this the right way, feel free to let me know if you'd like a different direction, like a  WriterOption

After that first pass, it seemed like overkill to make it available to readers, so I made a new PR adding a WriterOption that feels a little better. Again, let me know your thoughts on a preferred implementation, or if you even think this should be an option.

I left a review on the PR you submitted, the approach sounds good to me 👍


No files in Root container contents for True Default Store file

I've used mfcmapi successfully in the past, but someone else requested a read receipt and I must have accidentally hit send. I cannot get outlook 2016 messages to send any more. They sit in my outbox. When I run mfmapi (32-bit), I can see the line with the Default Store column set to True. But when I open contents table in the Root Container, everything is empty. No files. There was one there earlier that I deleted permanently when the Abort submit button did nothing. Now, nothing shows. Even the hidden folder is empty. Help!

The root container of a mailbox would rarely ever have anything in it. It sounds like you need to expand the root container to see the items underneath it.

My usual practice is to follow the instructions that say to open the contents table in the Root container and see what's there. It's always worked in the past. I did look at items under the root container. This morning, though it seems to be working, so I'll leave it alone for now.


TimeoutError: Timeout waiting for element to be visible

const {Agent,XPathResult} = require('secret-agent');

process.env.SA_SHOW_BROWSER = 'true';

(async () => {

  const agentConfig = {
    name: `Agent-example-1`,
    showReplay: true,
    sessionsDir: 'session',
  };

  const agent = await new Agent(agentConfig);

  await facebook(agent);

})();

async function facebook(agent){
	await agent.goto('https://facebook.com/');

	await agent.waitForPaintingStable();

	console.log(`Domain loaded - ${await agent.document.title}`);

	await agent.waitForElement(agent.document.querySelector('button[data-cookiebanner="accept_button"]'), { waitForVisible: true, timeout: 4 });

	const button = await agent.document.querySelector('button[data-cookiebanner="accept_button"]');

	await agent.click(button);
}

The timeout variables should be timeoutMs and in milliseconds, but that means it should have just timed out after 30 seconds. Is the element "visible"? If you're wanting to just "see" if it's visible for up to 4 seconds, you need to try/catch around it.

element is visible, it's a cookie popup, tried all ways around it, did not work

Same problem with other sites, seems that ones using js frameworks are affected, secret agent do not see that popups are open and visible to the user

after extended testing I found out that xpath works, it can select the element.

await this.agent.waitForElement(result.singleNodeValue, { timeout: 5000 });

And then I can click on element by using result.singleNodeValue.click();
But it then executes this javascript based click that does not allow popups to be opened. (facebook login button)

If you try a waitForElement (without waitForVisible) and then you do a getComputedVisibility on the item, does it tell you more about why it thinks the element isn’t visible?

On Aug 13, 2021, at 7:54 PM, Jānis Blaus @.***> wrote:

after extended testing I found out that xpath works, it can select the element.
await this.agent.waitForElement(result.singleNodeValue, { timeout: 5000 });
And then I can click on element by using result.singleNodeValue.click();
But it then executes this javascript based click that does not allow popups to be opened. (facebook login button)
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

I made sure element is really there and there are no hidden duplicated by opening the console and running
document.querySelector('button[aria-label="Log in with Facebook"]') it works fine and selects the right element.
getComputedVisibility gives me this:
{
  isVisible: false,
  nodeExists: true,
  isConnected: true,
  hasContainingElement: true,
  hasCssVisibility: true,
  hasCssDisplay: true,
  hasCssOpacity: true,
  isUnobstructedByOtherElements: false,
  obstructedByElementId: 689,
  boundingClientRect: { y: 382.671875, x: 544, height: 54, width: 352, tag: 'button' },
  hasDimensions: true,
  isOnscreenVertical: true,
  isOnscreenHorizontal: true
}

So it looks like secret agent thinks there's an element on top of this button that covers it up. Is there something that covers the button that could be going away on hover in your browser?

There are no other elements covering it.
Spent hours on this and still cant find how to solve it.
Ive attached a very short code example in my first message.

@janisblaus Can you send over your session db for this one?

Sure, attaching it.
sessions.zip

This session looks like it actually didn't surface a bad query selector. If I re-run your original script it's working correctly for me. Can you send that session?

Sorry, after so much testing, I probably added the wrong file :(
I recreated everything again.

ced8ee30-084d-11ec-8359-8d6faae08c67.zip

This issue is still there @blakebyrnes
Tested with secret agent 1.5.13 on windows 10.
Here is a simplified example:
const {Agent,XPathResult} = require('secret-agent');

process.env.SA_SHOW_BROWSER = 'true';

(async () => {

  const agentConfig = {
    name: `Agent-example-1`,
    showReplay: true,
    sessionsDir: 'session',
  };

  const agent = await new Agent(agentConfig);

  await youtube(agent);

})();

async function youtube(agent){
	await agent.goto('https://www.youtube.com/');

	await agent.waitForPaintingStable();

	console.log(`Domain loaded - ${await agent.document.title}`);

  const button  = agent.document.querySelector('[aria-label="Agree to the use of cookies and other data for the purposes described"]');
  const button2 = agent.document.querySelector('ytd-thumbnail:first-child');

  //console.log(await agent.getComputedVisibility(button))

  try {
    console.log('Trying to locate and click terms button')
    await agent.waitForElement(button, { waitForVisible: true });
    await agent.click(button);

    console.log('Trying to locate and click first thumbnail')
    await agent.waitForElement(button2, { waitForVisible: true });
    await agent.click(button2);

  }catch(e){
    console.log('Could not click the button', e)
    console.log(await agent.getComputedVisibility(button))
    console.log(await agent.getComputedVisibility(button2))
  }
}

getComputedVisibility:
(first isVisible: false is correct because button is clicked correctly, problem is with button2 )
{
  isVisible: false,
  nodeExists: true,
  isConnected: true,
  hasContainingElement: true,
  hasCssVisibility: true,
  hasCssDisplay: true,
  hasCssOpacity: true,
  isUnobstructedByOtherElements: false,
  obstructedByElementId: 408,
  boundingClientRect: { y: 0, x: 0, height: 0, width: 0, tag: 'tp-yt-paper-button' },
  hasDimensions: false,
  isOnscreenVertical: false,
  isOnscreenHorizontal: false
}
{
  isVisible: false,
  nodeExists: true,
  isConnected: true,
  hasContainingElement: true,
  hasCssVisibility: true,
  hasCssDisplay: true,
  hasCssOpacity: true,
  isUnobstructedByOtherElements: false,
  obstructedByElementId: 6331,
  boundingClientRect: { y: 756, x: 996, height: 44, width: 78, tag: 'ytd-thumbnail' },
  hasDimensions: true,
  isOnscreenVertical: true,
  isOnscreenHorizontal: true
}

Trace:
2021-11-03T00:59:37.428Z ERROR [C:\projects\youtube\worker\node_modules\@secret-agent\core\lib\Interactor] Interaction.click - mousePosition not in viewport after mouse moves to prepare for click. {
  'Interaction.mousePosition': [ 'document', [ 'querySelector', 'ytd-thumbnail:first-child' ] ],
  target: {
    nodeId: 671,
    nodeVisibility: {
      isVisible: false,
      nodeExists: true,
      isConnected: true,
      hasContainingElement: true,
      hasCssVisibility: true,
      hasCssDisplay: true,
      hasCssOpacity: true,
      isUnobstructedByOtherElements: false,
      obstructedByElementId: 6331,
      boundingClientRect: { y: 756, x: 996, height: 44, width: 78, tag: 'ytd-thumbnail' },
      hasDimensions: true,
      isOnscreenVertical: true,
      isOnscreenHorizontal: true
    },
    domCoordinates: { x: 1045, y: 783 }
  },
  viewport: {
    positionX: 15,
    positionY: 40,
    screenWidth: 1440,
    screenHeight: 900,
    width: 1424,
    height: 826,
    deviceScaleFactor: 1
  },
  context: { sessionId: '48059930-3c41-11ec-a34e-a9b23136b62c', frameId: 1 },
  sessionId: '48059930-3c41-11ec-a34e-a9b23136b62c',
  sessionName: 'agent-example-1'
}

Could not click the button Error: Element or mousePosition remains out of viewport after 2 attempts to move it into view
    at DefaultHumanEmulator.moveMouseAndClick (C:\projects\youtube\worker\plugins\default-human-emulator\index.ts:196:13)
    at DefaultHumanEmulator.playInteractions (C:\projects\youtube\worker\plugins\default-human-emulator\index.ts:75:11)
    at CorePlugins.playInteractions (C:\projects\youtube\worker\node_modules\core\lib\CorePlugins.ts:189:7)
------REMOTE CORE---------------------------------
    at Function.reviver (C:\projects\youtube\worker\node_modules\commons\TypeSerializer.ts:208:26)
    at JSON.parse (<anonymous>)
    at Function.parse (C:\projects\youtube\worker\node_modules\commons\TypeSerializer.ts:24:17)
    at WebSocket.<anonymous> (C:\projects\youtube\worker\node_modules\client\connections\RemoteConnectionToCore.ts:67:42)
    at WebSocket.emit (node:events:390:28)
    at Receiver.receiverOnMessage (C:\projects\youtube\worker\node_modules\ws\lib\websocket.js:983:20)
    at Receiver.emit (node:events:390:28)
    at Receiver.dataMessage (C:\projects\youtube\worker\node_modules\ws\lib\receiver.js:517:14)
    at C:\projects\youtube\worker\node_modules\ws\lib\receiver.js:468:23
    at C:\projects\youtube\worker\node_modules\ws\lib\permessage-deflate.js:308:9
------CONNECTION----------------------------------
    at new Resolvable (C:\projects\youtube\worker\node_modules\commons\Resolvable.ts:17:18)
    at Object.createPromise (C:\projects\youtube\worker\node_modules\commons\utils.ts:68:10)
    at RemoteConnectionToCore.createPendingResult (C:\projects\youtube\worker\node_modules\client\connections\ConnectionToCore.ts:328:31)
    at RemoteConnectionToCore.internalSendRequestAndWait (C:\projects\youtube\worker\node_modules\client\connections\ConnectionToCore.ts:253:43)
    at RemoteConnectionToCore.sendRequest (C:\projects\youtube\worker\node_modules\client\connections\ConnectionToCore.ts:156:17)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Object.cb (C:\projects\youtube\worker\node_modules\client\lib\CoreCommandQueue.ts:104:26)
    at Queue.next (C:\projects\youtube\worker\node_modules\commons\Queue.ts:82:19)
------CORE COMMANDS-------------------------------
    at Queue.run (C:\projects\youtube\worker\node_modules\commons\Queue.ts:35:19)
    at CoreCommandQueue.run (C:\projects\youtube\worker\node_modules\client\lib\CoreCommandQueue.ts:100:8)
    at CoreFrameEnvironment.interact (C:\projects\youtube\worker\node_modules\client\lib\CoreFrameEnvironment.ts:72:29)
    at Function.run (C:\projects\youtube\worker\node_modules\client\lib\Interactor.ts:47:21)
    at Agent.click (C:\projects\youtube\worker\node_modules\client\lib\Agent.ts:286:22)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at youtube (C:\projects\youtube\worker\youtube.js:38:5)
    at C:\projects\youtube\worker\youtube.js:15:3
[1f1f0470-3c41-11ec-ad68-d78f4269f981.zip](https://github.com/ulixee/secret-agent/files/7464805/1f1f0470-3c41-11ec-ad68-d78f4269f981.zip)

--------------------------------------------------
------agent-example-1-----------------------------
------1f1f0470-3c41-11ec-ad68-d78f4269f981--------
--------------------------------------------------

If I skip clicking the first button and only try to click the thumbnail, it gives element visible timeout.


Retreive Status Code

Is there any way to get the status code that Strava is sending to the library. That would be easier for us to decide whether the exception occuring for rate-limit or unauthorize user or record not found etc.

What about catching the exception stravalib.exc.RateLimitExceeded etc?


qpdf create pdf file success but open failed

I am using qpdf to create PDF files, but sometimes the generated PDF files fail to open because of the following figure

I've never seen this behavior before, but it's very likely that the cause is that the code in qpdf that converts numbers to strings is sensitive to locale settings in the caller's environment. If so, this would be a bug that I should fix. In the meantime, it would help if you send me the output of
printenv | grep LANG
printenv | grep LC_

Please check the output to make sure no sensitive information is disclosed before you post on the ticket. If you run qpdf in a shell. That might help me in reproducing this so I can be sure my fix is correct.
As a workaround, if you run qpdf in a shell where the LC_* environment variables are unset, do you still see the problem? What about on a completely clean system? It's also possible this is not based on the environment but based on something else on the system, or if you're calling libqpdf from a C++ program, something else in your application. The main thing I need to do is to ensure that my use of the library calls is protected against contamination from the outer environment.

The character set is en_US.UTF-8

I have find the bug, std::locale::global(std::locale("")); is added before using qpdflib library;So it leads to the format problem of qpdf library

Okay, thanks. I will have to look into exactly what this does and make sure that I can prevent it from messing up qpdf since the numeric conversion routines in qpdf are explicitly not intended to be locale-aware. In the mean time, my guess is that it would be sufficient to just ensure that, whatever this does, it is not in effect for the QPDFWriter::write() call. More likely than not, nothing else going on in the library should be adversely affected. That's not a guarantee since I've never tested it in this way. It's possible that it could affect error messages. Unfortunately qpdf is not properly internationalized, but there are numeric values in some error messages. But nothing should care about that.
I'll leave this ticket open as a reminder to protect qpdf against user-provided locale settings. Thanks for following up.

qpdf 10.0.2 (or whatever's next) will work correctly even if you call std::locale::global(std::locale("")); in your own code. I have updated the numeric conversion code to protect itself against the caller's locale. Thanks for bringing this to my attention. The idea that my numeric conversions were prone to this had completely escaped my notice.


APT whitelist request for cmake=3.2.3-5ppa5~precise1

Either this or forcing a newer version of cmake

Ran tests, but could not found source package. Either the source package for cmake=3.2.3-5ppa5~precise1 does not exist, or the package needs an APT source. If you wisht to add an APT source, please follow the directions on https://github.com/travis-ci/apt-source-whitelist#source-approval-process. Build results: https://travis-ci.org/travis-ci/apt-whitelist-checker/builds/72545671.


MUST be included the requirements

I reviewed the documentation but I didn't see the requirements for the installation (like laravel have). You must have this part on the installation so developers will know what versions of laravel, php, etc. needed. Thanks!

Voyager's readme file lists the supported laravel versions, as does our documentation.  However compiser.json is the source of truth for all dependency versions

Hi @fletch3555 . Thanks for the response. Can you give me a link for this?

I'll answer for fletch...
Access this page and scroll down until you see the readme text... https://github.com/the-control-group/
If you don't find, access this: https://github.com/the-control-group/voyager/blob/1.1/README.md

Closing as this seems to be solved.


Default system configuration file

Kairos version:

=2.5.0

Describe the bug
/etc/elemental/config.yaml was supported as a way to inject default system configs. As part of those we could configure branding menu to be applied during upgrade and reset, and additional cloud config path to look into the system.
Now we expect all that data to come from the user-data, however, we don't have a way to specify those settings when we build the images. The config collector scans configurations exclusively in paths that are either persistent or not accessible on build time (https://github.com/kairos-io/kairos-agent/blob/0119b9cfb264e9ddbdf91ca0b37f80a89059cea8/main.go#L39)
To Reproduce

Expected behavior
Have a canonical /etc/kairos/config.yaml where we collect configurations.
Why: it can be both ephemeral (dynamically written in the cloud config file, see for example jetson nano configuration here: https://github.com/kairos-io/packages/blob/27b86d41992d7a1deb76b2e2123f69ea2c7918c3/packages/static/kairos-overlay-files/files/system/oem/12_nvidia.yaml#L36 ) or static (if shipped by the OS image)
Logs

Additional context

@mudler I have a couple of questions:

now with the collector we think more of directories, would it make sense to just say everything  under /etc/kairos? or does it need to be a specific file?
do we need to stay backwards compatible in 3.0? so /etc/elemental/config.yaml and /etc/kairos/config.yaml or just the latter?
can you clarify the point of being ephemeral or static? is it just that because it is created during initramfs stage vs it being an overlay file, or what do you mean?

If that's an option, I would suggest a directory that will be used only for configs. That's because when we use directories that have lots of irrelevant files in them, a huge debug log is created with the reasons for skipping. I think that's the case with one of the directories we currently scan.

@mudler I have a couple of questions:
1. now with the collector we think more of directories, would it make sense to just say everything  under `/etc/kairos`? or does it need to be a specific file?

This sounds fine :+1:

2. do we need to stay backwards compatible in 3.0? so `/etc/elemental/config.yaml` and `/etc/kairos/config.yaml` or just the latter?

I'd keep the compatibility for 3.0 - and mark it in the release note that we are not going to read /etc/elemental/config.yaml from the next versions.

3. can you clarify the point of being ephemeral or static? is it just that because it is created during `initramfs` stage vs it being an overlay file, or what do you mean?

What I mean is that by supporting /etc/kairos as path being scanned we can:

Generate on-the-fly a /etc/kairos/config.yaml with cloud config before boot (for example as we used for injecting jetson nano defaults when the initramfs stage was being run)
Add a file /etc/kairos/config.yaml in the OS image that will take effect (while building the OS and preparing a downstream derivative)


Blog app: add API endpoints

Build an API that is RESTful.
Understand importance of APIs and docs following standards.
Create an API endpoint to list all posts for a user.
Create an API endpoint to list all comments for a user's post.
Create an API endpoint to add a comment to a post. Remember that the owner of the comment is the user that makes it; it should not be set by passing a parameter.
Your API endpoints should receive JSON and respond JSON as well.

Hello @Deepakdanger,
Thank you


PageFault error code now reports page presence correctly

If page directory was present and ring3 access or write protection violation caused page fault on page directory level, page presence was not checked and it was assumed to be present.
This PR adds this check to correctly report present bit in page fault error codes.
I found this bug as my OS uses the present bit in error code of page fault as the initial requirement for demand paging.
This is my first time writing rust, so please tell me if something should be done differently.

Interesting find, this is a pretty serious bug.

This is my first time writing rust, so please tell me if something should be done differently.

I think you could avoid the extra function by moving the write/user checks down and combining them with the page table checks. I'll leave a suggestion in the review.
I'll try to write some tests for this.

Done.

Thanks!


Implement APIs to support proof-of-concept proposal module (M2)

In order to complete Milestone 2 for Catalyst Voices, we must satisfy the following item:
“Deploy minimally functional proposal module to testnet for refinement over course of project”
This ticket implements the required backend functionality to support #245
As a user, I need to be able to save and update proposal information to the Catalyst backend, so that I can edit them later, as well as have them discovered by other users.
AC:

Application can fetch proposal template JSON
Application can read existing proposals
Application can create new proposals
Application can update existing proposals

@minikin please add initial data schema specs already defined

@minikin can you please clarify the approach to meet the requirements here? Does this ticket still need to be groomed, or will this be fully replaced by the remote widget PoC outlined in this other ticket?
https://github.com/input-output-hk/catalyst-voices/issues/254

UPDATE: full feature APIs not needed to satisfy M2. The PoC Proposal Module will make use of in-memory storage inside the browser, in order to simplify the delivery, and dedicate time / resources to more urgent demands. APIs will be delivered as part of the feature complete proposal submission & commentary currently scheduled for M5.


[BEAM-2241] Mark PipelineVisitor and AppliedPTransform as internal

cc: @davorbonaci

R: @aaltay

LGTM

Merged. Please manually close the PR.


Making it easier for people to know they need Entrez Direct.

I didn't have Entrez Direct installed, so I was getting a bunch of empty files and no error messages.
I fixed the script to stop if the Entrez Direct tools don't work (or anything else bad happens) and added a note about the dependency.
Also, I was only running the scripts in the first place because the Arvados dataset seems to itself be a collection of empty files today. I know that that wasn't the case before, since I peeked at the FASTAs there and they had data. Has the dataset expired or something?

@adamnovak  Apologies for the delay, I hadn't set up my github notifications correctly. Thanks for the bug report & pull request (merged the latter) and sorry about all the issues you've run into.
I created new URLs and updated my Readme.md, and also filed a bug report on Arvados:
https://arvados.org/issues/5915.
tl;dr I'm not sure why the link gives you 0 byte files, but trying the new links

The new links seem to work. Thanks!
On Tue, May 5, 2015 at 2:26 PM, nouyang-curoverse<EMAIL_ADDRESS>wrote:

@adamnovak https://github.com/adamnovak Apologies for the delay, I
hadn't set up my github notifications correctly. Thanks for the bug report
& pull request (merged the latter) and sorry about all the issues you've
run into.
I created new URLs and updated my Readme.md, and also filed a bug report
on Arvados:
https://arvados.org/issues/5915.
tl;dr I'm not sure why the link gives you 0 byte files, but trying the new
links
—
Reply to this email directly or view it on GitHub
https://github.com/nouyang-curoverse/GA4GH_regions/pull/2#issuecomment-99227668
.


setState warning using v5.0.1 with redux-form

Similar to the issue  #564 .
Just like in the above mentioned issue, I have a hidable form. The difference is: I have a ParentContainer that passes the visible prop to the ConnectedParent. Yet everything is fine unless I hide the Form dispatching asynchronously. In this case I get a warning:

setState(...): Can only update a mounted or mounting component. This usually means you called setState() on an unmounted component. This is a no-op. Please check the code for the Connect(Form(Form)) component.

I'm sorry, but i can't provide a clonable example. Some code to reproduce this bug:
import React, { Component } from 'react';
import {connect} from 'react-redux';
import {Field, reduxForm} from 'redux-form';

const Form = () => <Field type="text" component="input" name="field" />;
const ReduxForm = reduxForm({ form: 'example' })(Form);

class Parent  extends Component {
  componentWillReceiveProps(newProps) {
    if (this.props.visible !== newProps.visible) {
      if (!newProps.visible) {
        newProps.dispatch({ type: 'ANYTHING' });
      }
    }
  }

  render() {
    return
{this.props.visible ? <ReduxForm /> : null}
;
  }
}
const ConnectedParent = connect()(Parent);

const ParentContainer = ({ dispatch, visible }) => (

    <ConnectedParent visible={visible} />
    <button type="button" onClick={() => { dispatch({ type: 'SHOW_FORM' });} }>show</button>
    <button type="button" onClick={() => { dispatch({ type: 'HIDE_FORM' });} }>hide</button>
    <button type="button" onClick={() => { setTimeout(() => { dispatch({ type: 'HIDE_FORM' }); }, 0)} }>hide with timeout</button>

);

const mapStateToProps = state => ({ visible: state.app.form });
const ConnectedParentContainer = connect(mapStateToProps)(ParentContainer);

export default ConnectedParentContainer ;

Corresponding reducer:
const initialState = {
  form: false,
};

export default (state = initialState, action) => {
  switch (action.type) {
    case 'SHOW_FORM':
      return Object.assign({}, state, {
        form: true,
      });
    case 'HIDE_FORM':
      return Object.assign({}, state, {
        form: false,
      });
    default:
      return state;
  }
};

Use the ConnectedParentContainer component and the reducer in your app.
Steps to reproduce:

Click "show" button.
Click "hide with timeout" button.
Experience the Warning.

@osigum Can you provide more details? What does you store creation look like? What does your top-level component look like? Are you using the latest versions of react/redux-form/etc? There's not quite enough here to troubleshoot without making assumptions.

I was able to repro the bug above. I refactored out redux-form and ended up with this:
import React, { Component } from 'react';
import ReactDOM from 'react-dom';
import { Provider, connect } from 'react-redux';
import { createStore } from 'redux';

const store = createStore((state = { visible: false }, action) => {
  switch (action.type) {
    case 'SHOW_FORM': return { visible: true };
    case 'HIDE_FORM': return { visible: false };
    default: return state;
  }
});

const AAA = () =>
Hello
;
const BBB = connect(state => state)(AAA);

class CCC extends Component {
  componentWillReceiveProps(newProps) {
    if (this.props.visible && !newProps.visible) {
      newProps.dispatch({ type: 'ANYTHING' });
    }
  }

  render() {
    return
{this.props.visible ? <BBB /> : null}
;
  }
}
const DDD = connect()(CCC);

const EEE = props => <DDD {...props} />;
const FFF = connect(state => state)(EEE);

ReactDOM.render(
  <Provider store={store}>
    <FFF />
  </Provider>,
  document.getElementById('root')
);
store.dispatch({ type: 'SHOW_FORM' })
store.dispatch({ type: 'HIDE_FORM' });

It appears #579 (fixed #577) also fixes this one.


Add Readme for Coding Challenge 50.2 (Circle Packing) with Link to own Variation

By: romibi , Repository

Fade out circles after time and use camera...

Yay! Thank you for this!


[ARM64] PowerRename

Summary of the Pull Request
What is this about:
Adding configuration to PowerRename to ensure it builds correctly for ARM64
What is included in the PR:

How does someone test / validate:
Set configuration to Release|ARM64, set PowerRenameUI as the startup project, build & run.
Quality Checklist

[X] Linked issue: #490
[X] Communication: I've discussed this with core contributors in the issue.
[ ] Tests: Added/updated and all pass
[ ] Installer: Added/updated and all pass
[ ] Localization: All end user facing strings can be localized
[ ] Docs: Added/ updated
[ ] Binaries: Any new files are added to WXS / YML

[ ] No new binaries
[ ] YML for signing for new binaries
[ ] WXS for installer for new binaries

Contributor License Agreement (CLA)
A CLA must be signed. If not, go over here and sign the CLA.

🔥🔥🔥


Restructure modules

Summary of the changes / Why this improves CrateDB
Relates to https://github.com/crate/crate/issues/9888
This introduces a libs folder for "library" style components.
Rough plan:

[x] Move shared → libs/shared
[x] Move dex → libs/dex
[x] Move pgwire → libs/pgwire
[x] Split out components from core that are not plugin/module related and move them into shared
[ ] Move collections / utils stuff from es-server and es-core into shared
[x] Move dns-discovery         → plugins/
[x] Move azure-discovery       → plugins/
[ ] Move es-analysis-common    → plugins/ (or libs?)
[ ] Move es-analysis-phonetic  → plugins/
[ ] Move es-core               → libs / integrate into libs/shared
[x] Move es-discovery-ec2      → plugins
[ ] Move es-plugin-classloader → plugins ?
[x] Move es-repository-azure   → plugins
[x] Move es-repository-hdfs    → plugins
[x] Move es-repository-s3      → plugins
[x] Move es-repository-url     → plugins
[ ] Move es-transport          → libs ?
[ ] Move es-x-content          → libs

Checklist

[x] Added an entry in CHANGES.txt for user facing changes
[x] Updated documentation & sql_features table for user facing changes
[x] Touched code is covered by tests
[x] CLA is signed
[x] This does not contain breaking changes, or if it does:

It is released within a major release
It is recorded in CHANGES.txt
It was marked as deprecated in an earlier release if possible
You've thought about the consequences and other components are adapted
(E.g. AdminUI)

@seut Marked this ready-for-review to keep the PRs in a reasonable size. Will continue with some of the tasks outlined in a follow up PR.


Bug 2076671: Auto increase volume size to 20 GiB in the default storage class

Chery-pick of https://github.com/openshift/alibaba-disk-csi-driver-operator/pull/25 to 4.10
This will re-create the default storage class with a new volumeSizeAutoAvailable parameter that tells the driver to bump the volume sizes to 20 GiB.
Users may opt-out by creating a different default storage class.
cc @openshift/storage

/label backport-risk-assessed

/retest-required

/retest

I can see only some static pod failures and these pods don't use storage
static pod lifecycle failure - static pod: "etcd" in namespace: "openshift-etcd" for revision: 6 on node: "ci-op-wzn82w9v-57b6d-z2j88-master-0" didn't show up, waited: 2m30s

/retest

/label cherry-pick-approved


Build a timer feature

As a user, I want to be able to keep track of when tests are ready to be read with a timer in the app
Each card in the test queue should have a timer that can run for 15 minutes

Needs Mockup

Dupe of prime-central #121


what do i add

a

what if we made a polling solution to make starting polls to modify the server easier

do you speak Perl

Ohh like with reactions?
On Mon, Jan 29, 2024 at 6:05 PM Arch Linuxer @.***>
wrote:

what if we made a polling solution to make starting polls to modify the
server easier
—
Reply to this email directly, view it on GitHub
https://github.com/gradetools/parliament-bomb/issues/13#issuecomment-1915867434,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A6CGRMLMWQJAJYLYBIUS5VDYRBBOXAVCNFSM6AAAAABCQIMYRKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJVHA3DONBTGQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

Add a JSON configuration file and log file parser. It seems that Perl would be a good option for a system like this.

Ehat kinds of configurations?
On Tue, Jan 30, 2024 at 9:00 PM bolt9200 @.***> wrote:

Add a JSON configuration file and log file parser. It seems that Perl
would be a good option for a system like this.
—
Reply to this email directly, view it on GitHub
https://github.com/gradetools/parliament-bomb/issues/13#issuecomment-1918339299,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A6CGRMLGYUSEMAH25CFV6JLYRG6WTAVCNFSM6AAAAABCQIMYRKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJYGMZTSMRZHE
.
You are receiving this because you authored the thread.Message ID:
@.***>


(SERVER-2801) Use extension processing method directly

The type checking when getting extensions out of a CSR is more
restrictive than in bouncy caste's own functions. This commit updates us
to use their getInstance method for extensions, which is less brittle.
Our type checking breaks when updating to BC 1.63, which switched from
using DER types to using DL types when parsing pretty much everywhere.

For refernce: https://github.com/bcgit/bc-java/blob/master/core/src/main/java/org/bouncycastle/asn1/x509/Extension.java#L271-L283

The commit in BC that caused the trouble: https://github.com/bcgit/bc-java/commit/35983d13426db3451494efa04b95b27e369942a1

(nit) you wrote bouncy caste in the commit message


Issue 4

this issue is created on 7th nov, 2024

We will have a look at it

This issue has been automatically marked as stale due to inactivity. If no further activity occurs, it will be closed.

This issue has been closed due to inactivity.


Options Legend onClick Not working properly

Documentation Is:

[X] Missing or needed?
[X] Confusing
[X] Not sure?

Please Explain in Detail...
I am using Angular 12 ,"ng2-charts": "^3.0.9", and "chart.js": "^3.7.1", Legend onClick not working properly, sometimes works by click the legend and moving the cursor somewhere.
Example code is provided below and my requirement is when I click the Legend it will display the "Show Text" by checking the condition if isTransDocumentation  is true on the UI.
Html code

Total New Parameters

	Show Text

Angular code
public smeChartOptions: ChartOptions = {
	responsive: true,
	maintainAspectRatio: false,
	scales: {
		x: {
			stacked: true,
			display: true,
			title: {
				display: true,
				text: 'Category'
			},
		},
		y: {
			stacked: true,
			ticks: {
				// Include a dollar sign in the ticks
				callback: (value, index, ticks) => value + '%'
			}
		},
	},
	plugins: {
		/*stacked100: {enable: true},*/
		legend: {
			onClick: (e, item) => {
				console.log('I am in');
				this.isTransDocumentation = true;
			}
		}

	}
};

Your Proposal for Changes
Not Sure
Example
No response

Without a test case, I'm inclined to believe this an angular / ng2-chartjs problem

Closing due to lack of test case


Make Data NodeConvertible

@Chylis the reasoning behind not doing this previously was because some people (me in particular ;) ) might want to map their data as JSON to a full Node object.
I'm happy to be overruled on this, but would like to gather some opinions on it before going through. If that happens, let's throw on a few quick tests and get it merged.
Thanks for contributing to the Vapor community, we really appreciate it, and I look forward to some discussions on this thread.
For reference, this is my preferred implementation: https://github.com/LoganWright/Genome/blob/master/Sources/GenomeFoundation/Foundation.swift#L5
Which by implementing this here would break that lib.

Current coverage is 97.21% (diff: 100%)

Merging #23 into master will not change coverage

@@             master        #23   diff @@
==========================================
  Files            28         28
  Lines          1437       1437
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
  Hits           1397       1397
  Misses           40         40
  Partials          0          0

Powered by Codecov. Last update 4cf8c76...9538d55

@LoganWright I understand the issue and can imagine this can happen for converting any non-supported Node type to a Node. Even though, this addition could be nice, it might be better to leave these kind of implementations to projects using them, as each project could have a different interpretation of the conversions. You have a good example with Genome.Another option could be to simply rename your method, as

@LoganWright Thanks for the explanation (together with the Genome example). I can see the benefits of leaving it to the implementing projects.

@Chylis sounds good to me, thanks so much for participating in Vapor! I'm looking forward to more future contributions.


Latex not rendered in jupyter markdown

Latex equations are no longer rendered in Jupyter notebook markdown cells.
This is what a markdown cell looks like now.

Compared to in jupyter notebook:

This feature was working before the latest update.

Thanks for the bug. Can you include your VS code version and extension version information?
This seems to work for me in the latest version of VS code (1.59)


Fix test bugs

Fix #147

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:
see 1 file with indirect coverage changes


Syntax highlight design was blocked due to MIME type

🐛 Bug Report
If you define custom highlight design, you get the following error:
The resource from “http://localhost:3000/docs/css/atom-one-dark.css” was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff).
Have you read the Contributing Guidelines on issues?
👍
To Reproduce
(Write your steps here:)

Enable syntax highlighter in the config file.
Set up custom style and place it in the CSS folder
Link it in a config file.
Look into a console for errors.

Expected behavior
There should be a correct MIME type.
(Write what you thought would happen.)
Actual Behavior
Error:
The resource from “http://localhost:3000/docs/css/atom-one-dark.css” was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff).
(Write what happened. Add screenshots, if applicable.)
Your Environment

Docusaurus version used: 1.14.4
Environment name and version (e.g. Chrome 78.0.3904.108, Node.js 10.17.0): Node.js v10.15.3
Operating system and version (desktop or mobile): MacOS Catalina

Reproducible Demo
  highlight: {
    // Highlight.js theme to use for syntax highlighting in code blocks.
    theme: 'atom-one-light',
    defaultLang: 'js',
    themeUrl: 'css/atom-one-dark.css',
  },

More info: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options
(Paste the link to an example repo, including a siteConfig.js, and exact instructions to reproduce the issue.)

The Docusaurus team is actively working on Docusaurus v2.
After a very long alpha, Docusaurus v2 is now in beta.
We believe that the v2 beta is already better than v1, and v2 is ready for maintstream adoption.
Apart from critical issues and security fixes, we don't plan to invest much time on Docusaurus v1 anymore.
We are all closing v1 issues.
Maybe the current issue is already fixed in Docusaurus v2.
Please try to upgrade, and if the issue is still there on Docusaurus v2, open a new issue.


[Issue] Adding a symbol that would appear as one empty line

Information
Adding a symbol that would appear as one empty line will be convenient where you need to add a small empty separator.
Relevant log output
No response

There are already 2 symbols available with empty single row of pixels... Is that what you mean?

I didn't find different from space (32), what's the second one?

Check here:
https://github.com/trip5/Matrix-Fonts?tab=readme-ov-file#spaces
I'm not entirely certain these types of spaces work as intended so that's why I added:
https://github.com/trip5/Matrix-Fonts?tab=readme-ov-file#pixel-spaces-that-can-be-used-as-characters

Sorry! I was blind! As usual, the documentation has everything I need.
Thanks a lot :)


[AAP-24645] Instance Details Switch

Moved instance switch to instance details section so topology view side bar has access to it. Customers can enable and disable an instance from the instances list, instance detail page and topology sidebar now (Just added).
Related JIRA: AAP-24645

Made the requested changes


Handle less dependencies [question]

Hi,
I currently use https://github.com/diagramatics/plugin-less to load a less file in my project.
If I make changes to this file, the CSS do reload properly, but if I edit an imported file (by @import),
the hot reloading is not working. The server do send the files changes events.
I'm willing to take a look into it but I'm not sure where to start.
What would be the best approach to implement this?
Is there a way to tell the hot reloader that a module has external dependencies?
Maybe I should generate "noop" imports for all dependent files when generating the JS module?
Is there another loader that solves a similar problem?

At the moment the only approach is pretty hacky. But works well. Essentially, when you import a less file, the less plugin resolves and compiles that file (and all its imports), then hands it back to SystemJS. SystemJS then stores this in an internal registry (System.loads). Then when you edit the root less file on your computer, a change event is sent to systems-hot-reloader and it checks to see if its in System.loads. Since it is, it deletes is and asks the less compiler for the new version.
The problem now comes in when you edit one of the imported files. The change event gets sent to the browser. But SystemJS is only tracking the root file. So the event gets dropped.
The way i get around this is by storing (serverside) a list of all the less files (in my case postcss), then when any less file changes on the system, i send a change event for each root less file to the browser. This results in all root less file getting reloaded, including our change.
If you want an example of how I do this check out https://github.com/alexisvincent/jspm-devtools/blob/master/lib/index.js#L20.
I'm in the process of working on an extension to SystemJS to support these kinds of workflows.
https://github.com/alexisvincent/systemjs-hmr
Good luck. Let me know if you need anymore help.

On another note. The whole situation regarding devtools for SystemJS is going to be getting a whole lot better in a little bit. I'm working on something, and I know others in the community are as well.

Thanks I'll give your package a shot!
Can't wait to see how it will evolve, the jspm tooling is already pretty impressive as it is :).

I managed to include a similar function into my setup (was using https://github.com/piotrwitek/jspm-hmr).
I just created a little script that subscribe to the reloader events and I include it as a script tag in my dev index.html:
(function (exports) {
  function onChange(moduleName) {
    // ony continue if a less file changed
    if (!/\.less$/.test(moduleName)) return;

    // get list of SystemJS imported .less files
    var lessModules = Object.keys(SystemJS.loads)
      .map(function(key) { return SystemJS.loads[key]; })
      .filter(function(load) { return load.metadata.loader == "less"; })
      .map(function(load) { return new URL(load.address).pathname.slice(1); });

    // main module, don't need to reload
    if (lessModules.indexOf(moduleName) !== -1) return;

    // trigger reload on less modules
    lessModules.forEach(function(moduleName) {
      this.onFileChanged({ path: moduleName });
    }, this);
  }

  exports.setupLessHotReloader = function (reloader) {
    reloader.on('change', onChange.bind(reloader));
  }
})(window);

Then in the dev init script, I call it after creating the reloader:
// hot-reload config
SystemJS.import('systemjs-hot-reloader').then(function(HotReloader) {
  // if you're running server on custom port please remember to update below
  var reloader = new HotReloader.default('http://localhost:3000');
  setupLessHotReloader(reloader);
});
// load main module of your app with SystemJS
SystemJS.trace = true;
SystemJS.import('src/app');

Thanks for the help.

Awesome! Glad you got sorted. I wouldn't recommend jspm-devtools just yet. I'm actively developing it. But you might find it useful in the future. It traces the dependency tree server side and does all compilation server side, with cached results. Then when you load a page for the first time it takes 1-3seconds instead of the ridiculously long time it usually does. It achieves this by pushing the dependencies of a file either in a bundle or via HTTP2 server push. Then for hot reloading it only sends the change

No problem, yeah I took a look, it looks very promising to speed things up :).
I'll keep track of your progress and try it when it's stabilized.


Fix Company.luhn_algorithm and add missing tests.

There is some issues with Faker::Company.luhn_algorithm(number) :

Generates non "luhn-compliant" strings for even length number
Not tested

The problem was that luhn_algorithm didn't reverse the number submitted argument before splitting it (as it should).
This forgotten reverse resulted in the generation of non "luhn-compliant" strings for even length number.

index
0
1
2
3
4
5

odd_base
9
8
4
5
6

reserved_odd_base
6
5
4
8
9

selected numbers for odd_base *2 by luh_algorithm
9

4

6

right numbers for odd_base *2 by luh_algorithm
6

4

9

even_base
8
7
6
9
4
3

reserved_even_base
3
4
9
6
7
8

selected numbers for even_base *2 by luh_algorithm
8

6

4

right numbers for even_base *2 by luh_algorithm
3

9

7

No issue for odd length arguments ([9,4,6] vs [6,4,9]), only with even ([8,6,4] vs [3,9,7]).
This commit fixes the issue and add test_luhn_algorithm to avoid future issues.

I rebased on master since some new commits added conflicts since my PR has been submitted.
Can someone review this ?

Thanks for adding the missing test. I just asked you to rename the a variable because a doesn't mean anything in that context. I tested locally. Overall looks good. Thanks @01max

I improved the last commit.
Thanks for the review @vbrazo !


Fix #4872: Implement java.util.StringJoiner.

Updated. I also added a bunch of tests with blank elements (empty string elements that make the joiner non-empty).


Revert "Replication Optimization: Treating deleted put record's size same as delete  (#2297)"

This reverts commit dc51de39d882cc2a6d2c9c65166535a99d3cd91e.
We've found some bugs in this PR that makes GetRequests extremely huge and it breaks the server in replication. Now that the nonblocking network client would fix the slowness in cross-colo replication, we might as well just disable this feature by removing the code all together.

https://github.com/linkedin/ambry/pull/2297

Codecov Report
Base: 73.19% // Head: 53.56% // Decreases project coverage by -19.64% :warning:

Coverage data is based on head (2104039) compared to base (2107bd6).
Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files
@@              Coverage Diff              @@
##             master    #2364       +/-   ##
=============================================
- Coverage     73.19%   53.56%   -19.64%
+ Complexity    10713     7214     -3499
=============================================
  Files           767      767
  Lines         59639    59619       -20
  Branches       7310     7307        -3
=============================================
- Hits          43652    31934    -11718
- Misses        13586    24863    +11277
- Partials       2401     2822      +421

Impacted Files
Coverage Δ

...main/java/com/github/ambry/config/StoreConfig.java
97.50% <ø> (-0.05%)
:arrow_down:

.../messageformat/DeleteMessageFormatInputStream.java
57.89% <ø> (-33.78%)
:arrow_down:

...n/java/com/github/ambry/store/PersistentIndex.java
88.50% <100.00%> (+0.11%)
:arrow_up:

...in/java/com/github/ambry/router/CryptoService.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../java/com/github/ambry/clustermap/PartitionId.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...rc/main/java/com/github/ambry/utils/MockClock.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ava/com/github/ambry/config/JsonAccountConfig.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...java/com/github/ambry/utils/ByteBufferChannel.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ava/com/github/ambry/cloud/azure/AzureMetrics.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...a/com/github/ambry/rest/ServerSecurityService.java
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 402 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Update jsoniter-scala-core, ... to 2.13.11

Updates

com.github.plokhotnyuk.jsoniter-scala:jsoniter-scala-core
com.github.plokhotnyuk.jsoniter-scala:jsoniter-scala-macros

from 2.13.6 to 2.13.11.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.github.plokhotnyuk.jsoniter-scala" } ]

labels: library-update, early-semver-patch, semver-spec-patch, commit-count:1

Superseded by #316.


multiple RUN commands in dockerfile

Describe
Right now we have a lot of RUN commands in the dockerfile which is not a good practice because each RUN line adds a layer to the image, which increases the overall build time. Also, caching doesn't take place properly with multiple RUN commands.
Desired Solution
Multiple commands should be merged into one, wherever possible. This will decrease the overall build time and improve the caching process.  Also, I feel that the two lines RUN node --version and RUN npm --version are kind of unnecessary as they are not contributing in a good way to the actual build.
Reference article : stackoverflow

@Harkishen-Singh @muskankhedia @aquibbaig  I'd like to work on this issue if approved


pub broken

when i go to file -> publish menu, log in, and click "publish"
it opens a new window and redirects to
https://pub.beakernotebook.com/publications/featured#%2Fpublications%2Fe4f95c6a-9861-11e6-9fbd-e32d9c88b3a8
but it should be
https://pub.beakernotebook.com/publications/e4f95c6a-9861-11e6-9fbd-e32d9c88b3a8
the server changed the format for the URLs, the client needs to follow suit.

Actually It goes to this URL:
https://pub.beakernotebook.com/#/publications/505753de-99f4-11e6-9806-83a60007c496
but "pub.beakernotebook.com" redirects it to :
https://pub.beakernotebook.com/publications/featured#%2Fpublications%2Fe4f95c6a-9861-11e6-9fbd-e32d9c88b3a8
But I change URL to :
https://pub.beakernotebook.com/publications/505753de-99f4-11e6-9806-83a60007c496

and https://github.com/twosigma/beaker-notebook/pull/4854


add CI plans for mqtt app starter

CI plan added: https://jenkins.spring.io/job/spring-scst-app-starters-mqtt-master-ci/
@jvalkeal @sabbyanandan


Suggestion for a better example program

The format string in the printf_s explains what __umulh does better, visually speaking, by zero-padding all 64-bit values and separating the high and low parts of the product.

#sign-off


Datalabel of an AxisSection is not displayed correctly

How to reproduce?

Create a CartesianChart
Define an X-axis
LabelRotaion = -25
Define AxisSections for the X-axis
DataLabel = True

When hovering over the lables, a part of the label highlight is not visible (see screenshot)
Extra notes

I think that it's related to the calculation of the chart size.
I am using

Live-Charts version
0.9.7

.Net Version
2015

Windows
7

Also, the Y-axis is not displayed with the parameters: Position = RightBottom, IsMerged = true.
Position = RightBottom, IsMerged = false - is OK
Position = LeftTop, IsMerged = true - is OK

This is caused due the chart limit, the 'September' label fill exceeds the chart container, this container does not allow the chart to have content out of if, this fix requires a major review, since some features will be broken if we remove the Clip limit.

Same problem for me.
The Y-axis is not displayed with the parameters: Position = RightBottom, IsMerged = true.
Any fix is planned ?
I am using

Live-Charts version
Geared 1.2.9

.Net Version
4.7.1

Windows
Windows 10


replace old fashined markdown templates with forms

this will help the readability of issues a lot 🤓
Take a look at it yourself here and feel free to create a report to test it yourself, since this is the only purpose for this playground-repo 🙆🏻♂

BTW: This must be done to main since otherwise they would not appear....

Wow! I like it :-)


Rasa X Renaming models

Description of Problem: When uploading a model file via curl command to Rasa X, it checks that the model name not already exists. To upload an external updated model it needs to be renamed before on the client side. When training a model with Rasa X, it creates an autogenerated name timestampXYZ.tar.gz.
Overview of the Solution: Add functionality to rename a model in the UI and maybe also to allow override an existing model with the same name.
Examples: Helps to improve users flexibility when sharing models or to sort models.

Thanks for submitting this feature request 🚀@federicotdn will get back to you about it soon!✨

?

Hey @Ben-Mark, apologies for the delay. @TyDunn do you know if we have this feature planned for the future?

Hi @Ben-Mark. As long as I upload a model with a unique name to Rasa X, I am able to upload it with any name. Is that not the case for you?
Why would you want to override an existing model with the same name?

Hi @TyDunn. Overriding an existing model would allow easier interaction with Rasa X and other services (plain Rasa servers etc.) via Rasa X API.
Example for override: You have a running Rasa instance with a model myAwesomeModel loaded. Then you want to make some improvements to that model in Rasa X. You could just upload myAwesomeModel to Rasa X, improve it (train it in Rasa X) and fetch myAwesomeModel externally to reload it into your Rasa instance.
Example for renaming: If you manage two chatbot models with Rasa X myPizzaBot and myHotelBot. When both are uploaded to Rasa X and retrained, then you don't know if the new trained models belongs to myPizzaBot or myHotelBot due to the autogenerated name timestampXYZ.tar.gz.

Hi @neumann-d. I have a couple more questions to understand how you are using Rasa X 😄
Override: what advantages do you see in training a model in Rasa X instead of training it in a CI/CD pipeline (e.g. improve myAwesomeModel in Rasa X --> push changes in Rasa X UI to remote Git repo --> triggers a CI/CD pipeline that trains the model, naming it myAwesomeModel, and automatically uploads myAwesomeModel externally to reload it into your Rasa instance)?
Renaming: Why are using Rasa X for two assistants: myPizzaBot and myHotelBot?

Hi @TyDunn.
Normally when I use Rasa X, then I train it serveral times in Rasa X to directly test the changes. The new Git integration feature would be also a nice way to apply changes to another service, right. But I think not all users want to setup an additional Git Repo and implementing the option to override/renaming should not be that big deal.
Rasa X needs a lot of resources to boot up/run in a mircoservice environment. Therefore it is nice to only launch one instance for improving multiple assistants.

@TyDunn
Every model I train via rasa-x is saved in s3 (unfortunately with another node service hack that manages the i/o changes on the machine's disk since the s3 storage feature is not working, added a different issue for that.)
The problem is that I need to go all the way to s3 and to rename the models , simply to give them meaningful names.
that's a real pain because all the trained models via rasa-x get a date-time name which is unreadable
I wish I had the option to click on the model name and a rename capability would exist.
That way, I can focus on my NLP training and not go back and forth to my s3 models repo.

Thank you for raising this issue about Rasa X. We decided to stop supporting the Community Edition (free version) of ‘Rasa X’ (see more info here: https://rasa.com/blog/rasa-x-community-edition-changes/). That’s why we’ve closed your issue. We suggest that Rasa Enterprise customers raise the issue with our customer support team if they haven’t done so already.


fix support to insert the largeint in array

Proposed changes

This pr is used to support to insert the largeint in array.
Before the fix, we can't insert largeint  in array.
MySQL [(none)]> insert into example_db.array_test05 values(1, [117341182548128045443221445]);
ERROR 1105 (HY000): errCode = 2, detailMessage = Number out of range[117341182548128045443221445]. type: LARGEINT
3.After the fix, we can insert the largeint  in array successfully.
MySQL [(none)]> select * from example_db.array_test05;
+------+-------------------------------+
| k1   | k2                            |
+------+-------------------------------+
|    1 | [117341182548128045443221445] |
+------+-------------------------------+

Issue Number: #7570
Problem summary
Describe your changes.
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[x] No
[ ] I don't know

Has unit tests been added:

[ ] Yes
[x] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[x] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[x] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[x] No

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

LGTM


NuGet availability

Can you make Direct3DHook available on NuGet?

@Asconius currently the library is written as a starting point for further work. There are a number of things that would need to be done to make this a more reusable library in the form of a NuGet.
I'll leave this here as a request, but I think a fair bit of repo reorganisation is required first.


fumpt v3.3.x

Description
This applies gofumpt automatic code formatting to v3.3.x and I'd love it if we can get it merged before #1709

Before we can merge this PR, please make sure that all the following items have been
checked off. If any of the checklist items are not applicable, please leave them but
write a little note why.

[ ] Targeted PR against correct branch (see CONTRIBUTING.md)
[ ] Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
[ ] Code follows the module structure standards.
[ ] Wrote unit and integration tests
[ ] Updated relevant documentation (docs/) or specification (x/<module>/spec/)
[ ] Added relevant godoc comments.
[ ] Added a relevant changelog entry to the Unreleased section in CHANGELOG.md
[ ] Re-reviewed Files changed in the Github PR explorer
[ ] Review Codecov Report in the comment section below once CI passes

any time :).  Since it's approved now I can fumpt my branch too lol

@faddat Following the same reasoning as for #1709, I am going to close this PR as well, ok?


Feature/fix source change split

:tada: This PR is included in version 1.1.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


DPO with local fine-tuned llama 3.1 70b not progressing

this is config file:
# Model Arguments
model:
  _component_: torchtune.models.llama3_1.lora_llama3_1_70b
  lora_attn_modules: ['q_proj', 'v_proj', 'output_proj']
  apply_lora_to_mlp: False
  apply_lora_to_output: False
  lora_rank: 8  # higher increases accuracy and memory
  lora_alpha: 16  # usually alpha=2*rank
  lora_dropout: 0.0

tokenizer:
  _component_: torchtune.models.llama3.llama3_tokenizer
  path: ./dachi_new_model_70b_3.1-instruct/tokenizer.model

checkpointer:
  _component_: torchtune.training.FullModelHFCheckpointer
  checkpoint_dir: ./dachi_new_model_70b_3.1-instruct
  checkpoint_files: [
    model-00001-of-00030.safetensors,
    model-00002-of-00030.safetensors,
    model-00003-of-00030.safetensors,
    model-00004-of-00030.safetensors,
    model-00005-of-00030.safetensors,
    model-00006-of-00030.safetensors,
    model-00007-of-00030.safetensors,
    model-00008-of-00030.safetensors,
    model-00009-of-00030.safetensors,
    model-00010-of-00030.safetensors,
    model-00011-of-00030.safetensors,
    model-00012-of-00030.safetensors,
    model-00013-of-00030.safetensors,
    model-00014-of-00030.safetensors,
    model-00015-of-00030.safetensors,
    model-00016-of-00030.safetensors,
    model-00017-of-00030.safetensors,
    model-00018-of-00030.safetensors,
    model-00019-of-00030.safetensors,
    model-00020-of-00030.safetensors,
    model-00021-of-00030.safetensors,
    model-00022-of-00030.safetensors,
    model-00023-of-00030.safetensors,
    model-00024-of-00030.safetensors,
    model-00025-of-00030.safetensors,
    model-00026-of-00030.safetensors,
    model-00027-of-00030.safetensors,
    model-00028-of-00030.safetensors,
    model-00029-of-00030.safetensors,
    model-00030-of-00030.safetensors,
  ]
  recipe_checkpoint: null
  output_dir: ./tmp/Meta-Llama-3.1-70B-instruct-dpo
  model_type: LLAMA3
resume_from_checkpoint: False
save_adapter_weights_only: False

# Dataset and Sampler
dataset:
  _component_: torchtune.datasets.preference_dataset
  source: csv
  data_files: Combined/DPO/combined_dpo_examples.csv
  column_map:
    chosen: Output_e
    rejected: Output_b
  split: train
seed: null
shuffle: True
batch_size: 1
n_accumulate: 4

# Optimizer and Scheduler
optimizer:
  _component_: torch.optim.AdamW
  fused: True
  weight_decay: 0.05
  lr: 5e-4
lr_scheduler:
  _component_: torchtune.training.lr_schedulers.get_cosine_schedule_with_warmup
  num_warmup_steps: 100

loss:
  _component_: torchtune.rlhf.loss.DPOLoss
  beta: 0.1
  label_smoothing: 0

# Training
epochs: 1
max_steps_per_epoch: 1000
gradient_accumulation_steps: 8  # Use to increase virtual batch size
compile: False  # pytorch compile, set to true for better perf/memory

# Logging
output_dir: ./tmp/lora_dpo_output/
metric_logger:
  _component_: torchtune.training.metric_logging.DiskLogger
  log_dir: ${output_dir}
log_every_n_steps: 1
log_peak_memory_stats: True

# Environment
device: cuda
dtype: bf16
enable_activation_checkpointing: True  # True reduces memory
enable_activation_offloading: False  # True reduces memory`

Output is frozen on the message:
INFO:torchtune.utils._logging:FSDP is enabled. Instantiating model and loading checkpoint on Rank 0 ...

Hey @enqidu! Thanks for checking torchtune out : ))
I'd like to try reproduce the behaviour you're seeing here. Could you share your hardware configuration, and the command which you used to run the recipe?
Aside, I noticed:
n_accumulate: 4

In your config. Is this a custom recipe which you're finetuning with? I'm curious what you're using this parameter for : )

Hello @SalmanMohammadi
I'm running on ml.p4de.24xlarge instance - 8 NVIDIA A100
using lora_dpo_distributed recipe.
Regarding the
n_accumulate: 4
Most of the config is copied from llama 3_1 8b DPO config, I haven't explicitly checked each parameter.

Hey @enqidu. I tried reproducing this using your config on a 8xA100 node but wasn't able to. I'm using the nightly version of torchtune and the latest stable installation of torch
root@bc943fa118d5:~/torchtune# tune run --nnodes 1 --nproc_per_node 8 lora_dpo_distributed --config /workspace/config.yaml
Running with torchrun...
W1209 18:31:53.618000 16723 torch/distributed/run.py:793]
W1209 18:31:53.618000 16723 torch/distributed/run.py:793] *****************************************
W1209 18:31:53.618000 16723 torch/distributed/run.py:793] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
W1209 18:31:53.618000 16723 torch/distributed/run.py:793] *****************************************
INFO:torchtune.utils._logging:Running LoRADPORecipeDistributed with resolved config:

batch_size: 1
checkpointer:
  _component_: torchtune.training.FullModelHFCheckpointer
  checkpoint_dir: /workspace/Meta-Llama-3.1-70B-Instruct/
  checkpoint_files:
  - model-00001-of-00030.safetensors
  - model-00002-of-00030.safetensors
  - model-00003-of-00030.safetensors
  - model-00004-of-00030.safetensors
  - model-00005-of-00030.safetensors
  - model-00006-of-00030.safetensors
  - model-00007-of-00030.safetensors
  - model-00008-of-00030.safetensors
  - model-00009-of-00030.safetensors
  - model-00010-of-00030.safetensors
  - model-00011-of-00030.safetensors
  - model-00012-of-00030.safetensors
  - model-00013-of-00030.safetensors
  - model-00014-of-00030.safetensors
  - model-00015-of-00030.safetensors
  - model-00016-of-00030.safetensors
  - model-00017-of-00030.safetensors
  - model-00018-of-00030.safetensors
  - model-00019-of-00030.safetensors
  - model-00020-of-00030.safetensors
  - model-00021-of-00030.safetensors
  - model-00022-of-00030.safetensors
  - model-00023-of-00030.safetensors
  - model-00024-of-00030.safetensors
  - model-00025-of-00030.safetensors
  - model-00026-of-00030.safetensors
  - model-00027-of-00030.safetensors
  - model-00028-of-00030.safetensors
  - model-00029-of-00030.safetensors
  - model-00030-of-00030.safetensors
  model_type: LLAMA3
  output_dir: /workspace/tmp/Meta-Llama-3.1-70B-Instruct/
  recipe_checkpoint: null
compile: false
dataset:
  _component_: torchtune.datasets.stack_exchange_paired_dataset
device: cuda
dtype: bf16
enable_activation_checkpointing: true
enable_activation_offloading: false
epochs: 1
gradient_accumulation_steps: 8
log_every_n_steps: 1
log_peak_memory_stats: true
loss:
  _component_: torchtune.rlhf.loss.DPOLoss
  beta: 0.1
  label_smoothing: 0
lr_scheduler:
  _component_: torchtune.training.lr_schedulers.get_cosine_schedule_with_warmup
  num_warmup_steps: 100
max_steps_per_epoch: 1000
metric_logger:
  _component_: torchtune.training.metric_logging.StdoutLogger
  log_dir: /workspace/tmp/lora_dpo_output/
model:
  _component_: torchtune.models.llama3_1.lora_llama3_1_70b
  apply_lora_to_mlp: false
  apply_lora_to_output: false
  lora_alpha: 16
  lora_attn_modules:
  - q_proj
  - v_proj
  - output_proj
  lora_dropout: 0.0
  lora_rank: 8
n_accumulate: 4
optimizer:
  _component_: torch.optim.AdamW
  fused: true
  lr: 0.0005
  weight_decay: 0.05
output_dir: /workspace/tmp/lora_dpo_output/
resume_from_checkpoint: false
save_adapter_weights_only: true
seed: null
shuffle: true
tokenizer:
  _component_: torchtune.models.llama3.llama3_tokenizer
  path: /workspace/Meta-Llama-3.1-70B-Instruct/original/tokenizer.model

INFO:torchtune.utils._logging:Hint: enable_activation_checkpointing is True, but enable_activation_offloading isn't. Enabling activation offloading should reduce memory further.
DEBUG:torchtune.utils._logging:Setting manual seed to local seed 940935333. Local seed is seed + rank = 940935333 + 0
INFO:torchtune.utils._logging:metric logger is initialized.
INFO:torchtune.utils._logging:FSDP is enabled. Instantiating model and loading checkpoint on Rank 0 ...
INFO:torchtune.utils._logging:Instantiating model and loading checkpoint took 55.68 secs

Would you be able to share which versions of torchtune and torch you're using?
cc @ebsmothers if there's something obvious here

dataset:
component: torchtune.datasets.preference_dataset
source: csv
data_files: Combined/DPO/combined_dpo_examples.csv
column_map:
chosen: Output_e
rejected: Output_b
split: train
Might it be a dataset problem?

INFO:torchtune.utils._logging:FSDP is enabled. Instantiating model and loading checkpoint on Rank 0 ...

If you're hanging here the issue is in trying to instantiate the model, which happens before we setup data.
As a sanity check, could you try running the normal LoRA distributed recipe using this config - modifying the checkpoint and tokenizer directories to point to where you've downloaded your models?
If this also fails it may be something due to your setup.

Wow, seems like I'm having a similar problem with LoRA distributed recipe. Definitely, something is wrong with my setup but what could be the reason?
Thanks very much

try
pip uninstall torchao
pip install torchao
pip install -e .["dev"]

I'm not sure why, but in my case it worked.

try export NCCL_P2P_DISABLE=1, I double there is  no p2p in your gpus.

Thanks @yechenzhi !
Solving the problem has progressed so far.
Things I changed, I updated to new Torchtune version and fine-tuned llama-3.1-70b again.
Ran with TORCH_CPP_LOG_LEVEL=INFO NCCL_DEBUG=INFO
So far DPO started working, and the model is initialized as far as I can see.
Now I have a problem with the dataset, but this problem is solved for now.

OK, so I will write about another problem later but as I can see
DPO trainer doesn't save adapter_config.json file after training. I think it might cause some problems for users.

Regarding the main problem about output being frozen on the message:
INFO:torchtune.utils._logging:FSDP is enabled. Instantiating model and loading checkpoint on Rank 0 ...
As I can see, it only progresses if checkpoints are already cached. It has trouble reading from the disk and the problem persists.

@SalmanMohammadi @enqidu I'm not sure. Previously, I encountered the same issue and debugged it, finding it stuck here, I suspect the scatter operation requires P2P communication, but since the GPUs lack P2P support, it waits indefinitely. Setting export NCCL_P2P_DISABLE=1 allows PyTorch to use alternative communication methods between GPUs, though it significantly slows down performance. Disabling P2P enables the code to run, but at a much slower speed.

@yechenzhi I also experienced this behaviour on 2xRTX6000s. I found a couple related discussions (e.g. https://discuss.pytorch.org/t/ddp-training-on-rtx-4090-ada-cu118/168366, https://forums.developer.nvidia.com/t/standard-nvidia-cuda-tests-fail-with-dual-rtx-4090-linux-box/233202/59) - the issue seemed to be around P2P not being enabled for certain cards (Ada desktop cards)/bugs in older NVIDIA drivers. It's weird that @enqidu was seeing this on A800s, maybe something unique to their hardware setup.
I think disabling P2P communication or trying to update your drivers by following the discussion in the NVIDIA dev forum linked above might be one way to get around this. I'm going to resolve this issue for now. Thanks so much for your help @yechenzhi!

I just want to reiterate that export NCCL_P2P_DISABLE=1 didn't work for me. It was (and still is) a model initialization issue with checkpoint shards. I found a workaround solution to read the model and then have torchtune read it from cache.

I just want to address that export NCCL_P2P_DISABLE=1  didn't work for me. It was (and still is) a model initialization issue with checkpoint shards. I found a workaround solution to read the model and then have torchtune read it from cache.

Sorry, I misunderstood your earlier comment! Could you share a bit more here? What was your workaround solution?

@enqidu Could you run nvidia-smi topo -m and share the output with us?


[Feature Request] ENCFS Mount

Any chance for rclone support encfs as mount param.

This is a duplicate of #651 - can you add your support to that issue please?
Thanks


Change separator from UI during combine/separate doesn't have any effect

Space is still used no matter what separator is selected. Generated mapping XML file looks fine (strategy is defined), so the problem is probably not in UI.
        <Mapping xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Mapping" id="mapping.7287" strategy="Colon" mappingType="Combine">
            <InputField xsi:type="ns2:JavaField" name="sourceString" docId="io.atlasmap.qe.test.SourceMappingTestClass" index="0" path="/sourceString" fieldType="String"/>
            <InputField xsi:type="ns2:JavaField" name="sourceChar" docId="io.atlasmap.qe.test.SourceMappingTestClass" index="1" path="/sourceChar" fieldType="Char"/>
...

As we are running out of time for enhancements, we should make short term fix for this, i.e. get separator delivered to the strategy, without model change.

Oh actually Mapping.getDelimiter() is invoked and delivered to the strategy... it's UI's fault placing delimiter as a strategy.
It even should place delimiter=":" instead of ...="Colon"...

Also MultiSpace is not supported by runtime at all...


Exclude pkginfo 1.3.0 from installation.

Note that we only require pkginfo transitively from twine, so arguably we could be a bit more careful and simply not require twine in Travis, but for now this will do just fine.
Resolves the problem affecting #874. This will affect all our CI jobs until resolved, so let's merge this sharpish. @shazow @sigmavirus24 @haikuginger for review.

👍

It looks like a new version (1.3.2) just got pushed to PyPi that fixes the packaging issue.

I'm inclined to still want to merge this, but it's not desperate.

It's beneficial at least from the perspective of knowing that this will break things.

This just prevents people from ending up with broken development environments if they have out of date PyPI mirrors.

:custard:


GENERATED column support

I think this is a feature request... perhaps I should just try to contribute it myself, but thought I'd check in case support is planned already.
For a table with the following schema, datasette 0.51.1 doesn't pick up the GENERATED columns and the column list only contains (rowid, body)  If I edit the SQL and select the generated columns, it will happily show them.
At first glance it appears that def table_column_details(conn, table): would have to be refactored to use a different methodology to get the columns, since PRAGMA table_info(deeds); returns just 0|body|TEXT|0||0 so maybe it wouldn't be worth it.
CREATE TABLE deeds (
    body TEXT,
    id INT GENERATED ALWAYS AS (json_extract(body, '$.id')) STORED,
    consideration INT GENERATED ALWAYS AS (json_extract(body, '$.consideration')) STORED
);

I'm treating this as a bug - these columns should definitely be visible in Datasette.
I created my own test database using SQLite from Homebrew like this:
/usr/local/Cellar/sqlite/3.33.0/bin/sqlite3 deeds.db << EOF
CREATE TABLE deeds (
    body TEXT,
    id INT GENERATED ALWAYS AS (json_extract(body, '$.id')) STORED,
    consideration INT GENERATED ALWAYS AS (json_extract(body, '$.consideration')) STORED
);
INSERT INTO deeds (body) VALUES ('{
    "id": 1,
    "consideration": "This is the consideration"
}');
EOF

It looks like PRAGMA table_info skips "hidden" columns: https://www.sqlite.org/pragma.html#pragma_table_info
But PRAGMA table_xinfo does not: https://www.sqlite.org/pragma.html#pragma_table_xinfo

I poked around a little more.  The columns are included in table_xinfo() but this also includes other columns that might not want to be exposed.
sqlite> PRAGMA table_xinfo(deeds);
0|body|TEXT|0||0|0
1|id|INT GENERATED ALWAYS|0||0|3
2|consideration|INT GENERATED ALWAYS|0||0|3

Generated column support was added in SQLite 3.31.0, so any unit tests I write for this should use skipIf to only run on that version or later.

This change will also have an impact on how hidden virtual FTS tables are displayed, since apparently those have some hidden columns: https://latest.datasette.io/fixtures?sql=select+*+from+pragma_table_xinfo('searchable_fts')

cid
name
type
notnull
dflt_value
pk
hidden

0
text1

0

0
0

1
text2

0

0
0

2
name with . and spaces

0

0
0

3
searchable_fts

0

0
1

4
docid

0

0
1

5
__langid

0

0
1

@simonw thanks for investigating so quickly.  If it is undesirable to change that hidden behavior, maybe something like this is a suitable workaround:
SELECT * FROM pragma_table_xinfo('deeds') where hidden in (0,2);
0|body|TEXT|0||0|0
1|id|INT GENERATED ALWAYS|0||0|2
2|consideration|INT GENERATED ALWAYS|0||0|2

I'm OK exposing hidden columns, unless someone comes up with a pressing reason not to.

Ouch, the tests pass on my laptop but failed in CI: https://github.com/simonw/datasette/actions/runs/392367997
Lots of failures look like this:
ERROR: conn=<sqlite3.Connection object at 0x7f44f0494030>, sql = 'select rowid,  from facetable order by rowid limit 51', params = {}: near "from": syntax error

Note the select rowid, from... - so it looks like invalid SQL queries are being constructed maybe due to mis-detecting columns somehow.
I wonder why it didn't fail on my laptop?

Working on this in a pull request: https://github.com/simonw/datasette/pull/1117

Here's the problem: https://www.sqlite.org/changes.html#version_3_26_0

2018-12-01 (3.26.0)

Added PRAGMA table_xinfo that works just like PRAGMA table_info except that it also shows hidden columns in virtual tables.

CI is running 3.22.0.


Bugfix/chart tags color

Updates colour data consumption of tags on chart to adapt to new <Tag /> component API.

We have taken advantage of this branch to add a new functionality.  Now chart elements will remember the colour they have been assigned avoiding changing its colour when they are added or removed from the graph.
@j8seangel has magic  {...spreadTheWord}

Closed this PR as the changes has been split into different PR's targeting different branches.


Reduce package size

The package size balloned between 0.1.2 and 0.1.3 (and all after) becuase I added a large demo.gif to the README. I think I can host this gif outside of the project to remove that from the package size.

:tada: This issue has been resolved in version 0.2.4 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This issue has been resolved in version 0.2.4 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket: