[SPARK-21571][Scheduler] Spark history server leaves incomplete or unrea…

…dable history files
around forever.
Fix logic

checkForLogs excluded 0-size files so they stuck around forever.
checkForLogs / mergeApplicationListing indefinitely ignored files
that were not parseable/couldn't extract an appID, so they stuck around
forever.

Only apply above logic if spark.history.fs.cleaner.aggressive=true.
Fixed race condition in a test (SPARK-3697: ignore files that cannot be
read.) where the number of mergeApplicationListings could be more than 1
since the FsHistoryProvider would spin up an executor that also calls
checkForLogs in parallel with the test unless spark.testing=true configured.
Added unit test to cover all cases with aggressive and non-aggressive
clean up logic.
What changes were proposed in this pull request?
The spark history server doesn't clean up certain history files outside the retention window leading to thousands of such files lingering around on our servers.  The log checking and clean up logic skipped 0 byte files and expired inprogress or complete history files that weren't properly parseable (not able to extract an app id or otherwise parse...)  Note these files most likely appeared to due aborted jobs or earlier spark/file system driver bugs.  To mitigate this, FsHistoryProvider.checkForLogs now internally identifies these untracked files and will remove them if they expire outside the cleaner retention window.
This is currently controlled via configuration spark.history.fs.cleaner.aggressive=true to perform more aggressive cleaning.
How was this patch tested?
Implemented a unit test that exercises the above cases without and without the aggressive cleaning to ensure correct results in all cases.  Note that FsHistoryProvider at one place uses the file system to get the current time and and at other times the local system time, this seems inconsistent/buggy but I did not attempt to fix in this commit.  I was forced to change one of the method FsHistoryProvider.getNewLastScanTime() for the test to properly mock the clock.
Also ran a history server and touched some files to verify they were properly removed.
ericvandenberg@localhost /tmp/spark-events % ls -la
total 808K
drwxr-xr-x   8 ericvandenberg  272 Jul 31 18:22 .
drwxrwxrwt 127 root
-rw-r--r--   1 ericvandenberg    0 Jan  1  2016 local-123.inprogress
-rwxr-x---   1 ericvandenberg 342K Jan  1  2016 local-1501549952084
-rwxrwx---   1 ericvandenberg 342K Jan  1  2016 local-1501549952084.inprogress
-rwxrwx---   1 ericvandenberg  59K Jul 31 18:19 local-1501550073208
-rwxrwx---   1 ericvandenberg  59K Jul 31 18:21 local-1501550473508.inprogress
-rw-r--r--   1 ericvandenberg    0 Jan  1  2016 local-234
Observed in history server logs:
17/07/31 18:23:52 INFO FsHistoryProvider: Aggressively cleaned up 4 untracked history files.
ericvandenberg@localhost /tmp/spark-events % ls -la
total 120K
drwxr-xr-x   4 ericvandenberg  136 Jul 31 18:24 .
drwxrwxrwt 127 root           4.3K Jul 31 18:07 ..
-rwxrwx---   1 ericvandenberg  59K Jul 31 18:19 local-1501550073208
-rwxrwx---   1 ericvandenberg  59K Jul 31 18:22 local-1501550473508

Can one of the admins verify this patch?

Overall I like this as an option and the code looks good. Personally I use one log directory for all my logs (not just the SHS) so this wouldn't work for me, but I also run into dead files cluttering up my log dir and this would address that for larger servers.

QQ: Do we want to default turn on the feature in some future version? If so, what pre-condition in your mind should be fulfilled? Currently we have had too much configurations that default to turn off, this might lead common users hard to tune their Spark cluster.

This is something that should be defaulted to off, like I mentioned above many users like myself only use one log directory and when this is on it would delete any non-application logs in the log directory, including master and worker logs for example

Yea, I'm just thinking whether it is possible we can have a perfect approach that we can be confident to turn it on by default.

The default is off, so people can opt-in to more aggressive clean up.
Is this okay to be merged?

@ericvandenbergfb Could you please rebase this to the latest master so we can continue review it? Also cc @vanzin @jerryshao

@ericvandenbergfb please also fix the PR title, thanks.

See continuation of pull request at https://github.com/apache/spark/pull/19770


fix issue of data

fix the issues on not able to change user data
deplpyed endpoint for individual students

works fine


1.1.0 on Bintray gives 403

Hello, we're trying to deploy vector and getting a 403 from the bintray link that is referenced in the installation instructions for "deploying to a web server":
% wget https://dl.bintray.com/netflixoss/downloads/1.1.0/vector.tar.gz
--2017-07-25 15:51:23--  https://dl.bintray.com/netflixoss/downloads/1.1.0/vector.tar.gz
Resolving dl.bintray.com... <IP_ADDRESS>
Connecting to dl.bintray.com|<IP_ADDRESS>|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2017-07-25 15:51:23 ERROR 403: Forbidden.

Was this yanked from bintray? Is this no longer a supported installation method? Should we be using 1.1.2? It also results in a 403:
% wget https://dl.bintray.com/netflixoss/downloads/1.1.2/vector.tar.gz
--2017-07-25 15:52:19--  https://dl.bintray.com/netflixoss/downloads/1.1.2/vector.tar.gz
Resolving dl.bintray.com... <IP_ADDRESS>
Connecting to dl.bintray.com|<IP_ADDRESS>|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2017-07-25 15:52:19 ERROR 403: Forbidden.

Fwiw, we worked around this by making our own copy of the tarball in an S3 bucket so we could still install.

That's strange.
➜  ~ wget https://dl.bintray.com/netflixoss/downloads/1.1.0/vector.tar.gz
--2017-07-26 16:17:18--  https://dl.bintray.com/netflixoss/downloads/1.1.0/vector.tar.gz
Resolving dl.bintray.com... <IP_ADDRESS>, <IP_ADDRESS>
Connecting to dl.bintray.com|<IP_ADDRESS>|:443... connected.
HTTP request sent, awaiting response... 302
Location: https://akamai.bintray.com/ff/ff5a126b6d106c9f2d6e886f4a87a8708933dc997091ad13f3a7f1e65ebed5e2?__gda__=exp=1501111759~hmac=4f2342c518c05f8cf97fe905dfb217c418a2b8c4abf95eca1b690a2a0422bb24&response-content-disposition=attachment%3Bfilename%3D%22vector.tar.gz%22&response-content-type=application%2Fgzip&requestInfo=U2FsdGVkX1-U6eFK9UpbEWxsfTxqTPALKlIJlgJCqll10gmBL9OStHqH7GL5Ud6EbWp_oU_G-jNoI1Tk4l0icNPo0nEaNjAS-niu2x5LFeC14oZ3OcmJlsxZgT9YOzQHtuGAswOTeIrTTMTMybVgQw [following]
--2017-07-26 16:17:19--  https://akamai.bintray.com/ff/ff5a126b6d106c9f2d6e886f4a87a8708933dc997091ad13f3a7f1e65ebed5e2?__gda__=exp=1501111759~hmac=4f2342c518c05f8cf97fe905dfb217c418a2b8c4abf95eca1b690a2a0422bb24&response-content-disposition=attachment%3Bfilename%3D%22vector.tar.gz%22&response-content-type=application%2Fgzip&requestInfo=U2FsdGVkX1-U6eFK9UpbEWxsfTxqTPALKlIJlgJCqll10gmBL9OStHqH7GL5Ud6EbWp_oU_G-jNoI1Tk4l0icNPo0nEaNjAS-niu2x5LFeC14oZ3OcmJlsxZgT9YOzQHtuGAswOTeIrTTMTMybVgQw
Resolving akamai.bintray.com... <IP_ADDRESS>
Connecting to akamai.bintray.com|<IP_ADDRESS>|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2531005 (2.4M) [application/gzip]
Saving to: 'vector.tar.gz'

vector.tar.gz                           100%[===============================================================================>]   2.41M  --.-KB/s    in 0.1s

2017-07-26 16:17:19 (18.7 MB/s) - 'vector.tar.gz' saved [2531005/2531005]

Same goes for v1.1.0. Check the downloaded tarball and it looks good.
Could you check again @jtimberman . Might have been a temporary BinTray hiccup.

Closing due to inactivity.


Private key changed

Hello i restore my mobile i saved secret phrase when i put it again in zen wallet. Wallet opened again but there is no zen in my wallet also change private keys explain how yo regain access to zens

My mistake now i recovered


get_current_tid_inner for ios

I just confirmed and verified that pthread_threadid_np can indeed be called on ios. This has been compiled on my example

Closed as in favor of #37, which added CI check. Thank you anyway!


vertical space above fraction line is too small for \frac

source:
\left( x \right) \left( x^2 \right) \left( \frac{a}{b} \right) \left( \frac{a^2}{b} \right) \left( \dfrac{a}{b} \right) \left( \dfrac{a^2}{b} \right)

LaTeX (zoomed in so the size doesn't match exactly):

khan.github.io/KaTeX:

localhost:7936 (master):

Huh. When you take the screenshots, are you using the browser zoom feature or increasing the font size? Also, are those the same expressions? The first a/b looks like it's in displaystyle, while yours looks like it's in textstyle? (note that stuff on khan.github.io/KaTeX automatically adds \displaystyle to the generated code)

I didn't realize that \displaystyle was being added automatically.


Refining the helm chart section

Although the helm-chart for fluentbit-operator has now been upgraded to fluent-operator, there are some features that need to be improved. Reference can be made to：https://github.com/fluent/helm-charts
Some parameters need to be configured or customised to suit the situation.

Add monitoring section
Optional ha deployment for fluentd

Any other propsals?

autoscaling might not work for stateful set

autoscaling might not work for stateful set

Supported k8s verion > 1.9, see https://stackoverflow.com/questions/54663845/apply-hpa-for-statefulset-in-kubernetes

https://stackoverflow.com/questions/54663845/apply-hpa-for-statefulset-in-kubernetes

It would be a really nice feature to have and help even out Fluentd CPU usage, which is usually the bottleneck.
HPA now supports StatefulSets.

Anything I could do to help with the HPA support for fluentds deployed trough the operator?

Add monitoring section
Optional hpa deployment for fluentd
livenessProbe and readinessProbe

You can give your opinion and then we can discuss it together.


@types/openui5: MessageBox - actions property in mOptions parameter is missing

OpenUI5 version: 1.91.0
@types/openui5: 1.91.0
Steps to reproduce the problem:

What is the expected result?
No type error
What happens instead?
Type error due to missing property actions
Any other information? (attach screenshot if possible)
Probably the same error for warning(), information(), etc.

Hello @marcelschork
Thank you for sharing this finding. I've created an internal incident<PHONE_NUMBER>. The status of the issue will be updated here.
Regards,
Georgi

Hi @marcelschork
The fix will be included in the next release of the Typescript definitions library!
Regards,
Georgi


Fix compilation error for get_mut_or_tagged_slot function

Build in stable branch was failing with an error:
error[E0505]: cannot move out of `_` because it is borrowed
   --> src/slab.rs:460:13
    |
454 |     fn get_mut_or_tagged_slot(&mut self, id: Id<T>) -> Result<&mut T, Option<&mut TaggedSlot<T>>> {
    |                               - let's call the lifetime of this reference `'1`
...
457 |                 slot: Slot::Occupied { ref mut value },
    |                                        ------------- borrow of value occurs here
458 |                 tag,
459 |             }) if tag == id.tag => Ok(value),
    |                                    --------- returning this value requires that borrow lasts for `'1`
460 |             tagged_slot => Err(tagged_slot),
    |             ^^^^^^^^^^^ move out of value occurs here

Explanation of a similar problem can be found here.
This is the least messy fix that I could come up with.
Interestingly, there is no such problem with a get_or_tagged_slot function only because multiple non-mutable borrows of the same value are allowed.
BTW, should I bump the package version with this PR?

Awesome, thanks! On the "double runtime check" didn't mean perf-wise, just meant style-wise


Installation fails on development

I tried to install jumpscale with the docs on development: https://github.com/threefoldtech/jumpscaleX/tree/development/docs/Installation
#get the installer
curl https://raw.githubusercontent.com/threefoldtech/jumpscaleX/development/install/jsx.py?$RANDOM > /tmp/jsx
chmod 777 /tmp/jsx
#install
/tmp/jsx container-install

yet it fails with:
Connection to localhost closed.
development
Tue 11 11:11:44 InstallTools.py  -2683 - repos_get                          : get code:jumpscaleX:threefoldtech (['development'])
development
Tue 11 11:11:44 InstallTools.py  -1569 - checkoutbranch                     : Found branch: development
Tue 11 11:11:44 InstallTools.py  -2683 - repos_get                          : get code:digitalmeX:threefoldtech (['development'])
master
Tue 11 11:11:44 InstallTools.py  -1569 - checkoutbranch                     : Found branch: master
+ cd /sandbox/code/github/threefoldtech/digitalmeX
+ git checkout development -f
error: pathspec 'development' did not match any file(s) known to git.

EXCEPTION

        jsx                                                          : 552 : cli()
        /root/.local/lib/python3.6/site-packages/click/core.py       : 764 : return self.main(*args, **kwargs)
        /root/.local/lib/python3.6/site-packages/click/core.py       : 717 : rv = self.invoke(ctx)
        /root/.local/lib/python3.6/site-packages/click/core.py       : 1137: return _process_result(sub_ctx.command.invoke(sub_ctx))
        /root/.local/lib/python3.6/site-packages/click/core.py       : 956 : return ctx.invoke(self.callback, **ctx.params)
        /root/.local/lib/python3.6/site-packages/click/core.py       : 555 : return callback(*args, **kwargs)
        jsx                                                          : 281 : installer.install(sandboxed=False, force=force, gitpull=pull)
        code/github/threefoldtech/jumpscaleX/install/InstallTools.py : 2628: self.repos_get(pull=gitpull)
        code/github/threefoldtech/jumpscaleX/install/InstallTools.py : 2686: Tools.code_github_get(repo=sourceName, account=self.account, branch=self.branch, pull=pull)
        code/github/threefoldtech/jumpscaleX/install/InstallTools.py : 1588: raise RuntimeError("Could not checkout branch:%s on %s" % (branch, args["REPO_DIR"]))

Tue 11 11:11:44 core.py          - 762 -                                    : EXCEPTION: Could not checkout branch:['development'] on /sandbox/code/github/threefoldtech/digitalmeX
CANNOT CONTINUE
Connection to localhost closed.

EXCEPTION

        /tmp/jsx                                                     : 552 : cli()
        aleedhammam/.local/lib/python3.6/site-packages/click/core.py : 764 : return self.main(*args, **kwargs)
        aleedhammam/.local/lib/python3.6/site-packages/click/core.py : 717 : rv = self.invoke(ctx)
        aleedhammam/.local/lib/python3.6/site-packages/click/core.py : 1137: return _process_result(sub_ctx.command.invoke(sub_ctx))
        aleedhammam/.local/lib/python3.6/site-packages/click/core.py : 956 : return ctx.invoke(self.callback, **ctx.params)
        aleedhammam/.local/lib/python3.6/site-packages/click/core.py : 555 : return callback(*args, **kwargs)
        /tmp/jsx                                                     : 225 : docker.jumpscale_install(branch=branch, redo=reinstall, pull=pull, wiki=wiki)
        /tmp/InstallTools.py                                         : 3231: self.sshexec(cmd)
        /tmp/InstallTools.py                                         : 3080: Tools.execute(cmd2, interactive=True, showout=False, replace=False, asfile=True)
        /tmp/InstallTools.py                                         : 1057: original_command=command,
        /tmp/InstallTools.py                                         : 1064: res = Tools._execute_interactive(cmd=command, die=die, original_command=original_command)
        /tmp/InstallTools.py                                         : 483 : % (original_command, returncode)

Tue 11 13:11:44 core.py          - 762 -                                    : EXCEPTION: ***ERROR EXECUTE INTERACTIVE:
Could not execute:ssh -oStrictHostKeyChecking=no -t root@localhost -A -p 9122 'cd /tmp;python3 jsx install'
returncode:1

CANNOT CONTINUE

@waleedhammam could you retest the installer once https://github.com/threefoldtech/jumpscaleX/pull/620 is merged ?

@mdeecode followed https://github.com/threefoldtech/jumpscaleX/tree/development/docs/Installation  but it produced another error with
 - Installing jumpscaleX
bash: sudo: command not found

fixed on: https://github.com/threefoldtech/jumpscaleX/pull/637

i tried development branch with container-install install option and it worked correctly

now insystem option failed, and @waleedhammam  trying to solve it.

@Dina-Abd-Elrahman this issue for container installation only.
in system is in: https://github.com/threefoldtech/jumpscaleX/issues/602

we can close this issue now.


Campfire build instructions remain after built

Description

When building a campfire, you will have the build instructions remain after finishing it
Reproduction

Make campfire
Examine campfire
Screenshots

Additional context

What the scallop, I'll take a look at this when I get home


If format: revealjs and there are webr cells on a slide, slide should be scrollable by default

It's hard to predict how much space the output of a webr cell will be, so I'm finding myself adding the {.scrollable} class to every slide with a webr cell. And, of course, forgetting to do so every once in a while. It would be nice if the existence of a webr cell made the slide automatically scrollable. There may be reasons to not do this (either technical or not), but so far in my use-case, I can see it would be helpful.

Great idea, thanks!


Add more clarity about what happens to messages in mailboxes with legal hold when ZAP 'quarantines' malware.

https://learn.microsoft.com/microsoft-365/security/office-365-security/zero-hour-auto-purge#how-does-zap-affect-mailboxes-on-hold
"ZAP quarantines messages from mailboxes on hold."
This needs to include more information and be more explicit. Which messages is it quarantining? Later sentences talk about spam/phishing... I think this one is about malware, but it needs to say so if it is.
The link provided at the end includes no information about what happens when ZAP quarantines a message.
Most importantly, this paragraph should clearly state whether or not ZAPed messages with malware found in mailboxes with legal holds will still be discoverable - if not, eDiscovery folks need to clearly understand it. This paragraph can't leave that question unanswered.
Thanks!

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 4fb415a2-fc7c-5b9a-96de-f01be631bf07
Version Independent ID: c5d4ab93-4f59-a60f-5d42-1a6a52a37f4c
Content: Zero-hour auto purge in Microsoft Defender for Office 365
Content Source: microsoft-365/security/office-365-security/zero-hour-auto-purge.md
Service: microsoft-365-security
Sub-service: mdo
GitHub Login: @chrisda
Microsoft Alias: chrisda

To make it easier for you to submit feedback on articles on learn.microsoft.com, we're transitioning our feedback system from GitHub Issues to a new experience.
As part of the transition, this GitHub Issue will be moved to a private repository. We're moving Issues to another repository so we can continue working on Issues that were open at the time of the transition. When this Issue is moved, you'll no longer be able to access it.
If you want to provide additional information before this Issue is moved, please update this Issue before December 15th, 2023.
With the new experience, you no longer need to sign in to GitHub to enter and submit your feedback. Instead, you can choose directly on each article's page whether the article was helpful. Then you can then choose one or more reasons for your feedback and optionally provide additional context before you select Submit.
Here's what the new experience looks like.
Note: The new experience is being rolled out across learn.microsoft.com in phases. If you don't see the new experience on an article, please check back later.
First, select whether the article was helpful:

Then, choose at least one reason for your feedback and optionally provide additional details about your feedback:

Article was helpful
Article was unhelpful

Finally, select Submit and you're done!


Memory Leak Observed When Using OnvifClient and MediaClient Connections

I am experiencing a memory leak in my application when using OnvifClient along with MediaClient and Media2Client from the QuickNV.Onvif library. It appears that the OnvifClient does not implement IDisposable, which may be contributing to the issue. The memory leak is most noticeable when establishing new connections repeatedly, even though I attempt to close them using client.DeviceClient.CloseAsync() or client.DeviceClient.Abort().
Steps to Reproduce:

Create a new instance of OnvifClient for each connection to a camera.
Use MediaClient or Media2Client to perform operations.
After operations are complete, attempt to close the connection using client.DeviceClient.CloseAsync() or client.DeviceClient.Abort().
Repeat the process multiple times.
Observe memory usage increasing over time.

Code Sample:

public async Task<OnvifCameraDto> GetConfigurationForExistingCamera(string login, string password, string rtsp)
{
    var ip = ExtractIpFromRtsp(rtsp);
    var connection = await Connect(ip, login, password);

    OnvifCameraDto onvifCameraDto;

    try
    {
        onvifCameraDto = await GetConfigurationUsingMedia2Client(connection, ip);
    }
    catch (Exception ex)
    {
        Log.Error("Media2Client failed, falling back to MediaClient: " + ex.Message);
        onvifCameraDto = await GetConfigurationUsingMediaClient(connection, ip);
    }

    // Attempt to close the connection
    await connection.DeviceClient.CloseAsync();
    connection = null;

    return onvifCameraDto;
}

 private async Task<OnvifClient> Connect(string ip, string login, string password, int port = 80)
 {
     var client = new OnvifClient(
        new OnvifClientOptions
        {
            Host = ip,
            Port = port,
            UserName = login,
            Password = password
        });

     try
     {
         await client.ConnectAsync();
         return client;
     }
     catch (Exception ex)
     {
         throw new CustomException("Cannot connect to client", new List<string> { ex.Message }, statusCode: HttpStatusCode.NotAcceptable);
     }
 }

private async Task<OnvifRtspProfilesDto> GetRtspUrisForMainAndSubProfiles(string login, string password, string ip)
{
    var connection = await Connect(ip, login, password);

    OnvifRtspProfilesDto rtspProfiles;

    try
    {
        rtspProfiles = await GetRtspUrisUsingMedia2Client(connection);
    }
    catch (Exception ex)
    {
        rtspProfiles = await GetRtspUrisUsingMediaClient(connection);
    }

    // Attempt to close the connection
    await connection.DeviceClient.CloseAsync();
    connection = null;

    return rtspProfiles;
}

Observed Behavior:

Memory usage increases with each new connection.
Memory is not released after attempting to close connections using CloseAsync() or Abort().
Over time, this leads to high memory consumption and potential application instability.

Expected Behavior:

Memory used by connections should be properly released after closing them.
Repeated connections should not cause cumulative memory growth.

Additional Information:

MediaClient and Media2Client implement IDisposable, and I am using them within using statements to ensure proper disposal.
OnvifClient does not implement IDisposable, and there is no straightforward way to dispose of it.
The issue seems to originate from OnvifClient holding onto resources that are not being released.
I observe the largest memory leaks when establishing new connections with OnvifClient, even after attempting to close them.

Environment:

QuickNV.Onvif library version: 1.0.1
.NET version: .net 8
Operating System: Windows 11

Attempts to Resolve:

Used client.DeviceClient.CloseAsync() and client.DeviceClient.Abort() to close connections.
Ensured MediaClient and Media2Client are disposed properly using using statements.
Removed calls to Abort() as suggested in some recommendations, but the issue persists.

Possible Cause:

The lack of IDisposable implementation in OnvifClient might prevent proper resource cleanup.
Persistent connections or unmanaged resources within OnvifClient are not being released.

Request:
Could you please investigate this issue?
Consider implementing IDisposable in OnvifClient to allow for proper disposal of resources.
Provide guidance on how to correctly manage the lifecycle of OnvifClient instances to prevent memory leaks.
Any suggestions or workarounds to mitigate this issue would be greatly appreciated.
Thank you for your assistance!

Thanks for your feedback @Vesperino. OnvifClient class now implement IDispose interface.You can try version 1.0.2 at Nuget


adding support for X-EconomicAppIdentifier header

closes #5

Hi @dinesh - thanks for your contribution. Could you add it to the configuration example in README?

@ta corrected now.

@dinesh Had a single comment - please review.

@ta Right, fixed the typo :)

@dinesh Super - Thanks for your contribution!


Make it easy to add/remove a node in the same place

Circuit Construction Kit has this code:
    /**
     * Checks whether a child should be in the scene graph and adds/removes it as necessary.  This is to improve
     * performance so that the DOM only contains displayed items and doesn't try to update invisible ones.
     * @param inSceneGraph {boolean} - should the child be shown in the scene graph
     * @param parent {Node} - parent that contains the child in the scene graph
     * @param child {Node} - child added/removed from scene graph
     * @public
     * REVIEW: How about putting this in Node.js (we can discuss a potential name)?
     * REVIEW^(samreid): A general solution should keep track of the ordering--could we accomplish this in a robust way?
     */
    setInSceneGraph: function( inSceneGraph, parent, child ) {
      if ( inSceneGraph && !parent.hasChild( child ) ) {
        parent.addChild( child );
      }
      else if ( !inSceneGraph && parent.hasChild( child ) ) {
        parent.removeChild( child );
      }
    }

It would also be convenient to add/remove nodes for accessibility (as opposed to setVisible) so that the parallel dom is updated properly.
For PhET-iO we would like to add/remove nodes to reflow layouts, see https://github.com/phetsims/phet-io/issues/1204
For accessibility and phet-io layouts, it is also convenient to remove/add nodes
One possibility would be to add a parent node for each node that retains the place in the rendering order.
Another possibility could be to swap in a placeholder when a node is removed.
CCK will retain the sim-specific fix, but if scenery gets this functionality in the future, we should update CCK to use it.

It would also be convenient to add/remove nodes for accessibility (as opposed to setVisible) so that the parallel dom is updated properly.

This is now supported.


allow for empty navbar toggle breakpoint [bs4]

I wouldn’t mind, but the local tests didn’t run.
I’m a testing noob (for ember anyway) but i’ll give it a blind shot.

On 22 Sep 2017, at 14:48, Stephen Vance<EMAIL_ADDRESS>wrote:
@srvance requested changes on this pull request.
@basz https://github.com/basz Thank you very much! Good catch!
Would you mind creating a test for this case now that there's more complexity?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/kaliber5/ember-bootstrap/pull/451#pullrequestreview-64566668, or mute the thread https://github.com/notifications/unsubscribe-auth/AAIu3G2lx-n34wXY6fO-MC33ep7yBq26ks5sk6yngaJpZM4PgkIw.

Why didn't they run? Doesn't BOOTSTRAPVERSION=4 ember test work to run the tests under bs4? I don't have the right computer with me to give you exactly what I use. You can use --filter=navbar to reduce the set of tests during development.

that runs. thanks
i peeked at the travis cone and tried ember try:one ember-release-bs4, which fails with broccoli-tree could not be found, or something… nvm

On 22 Sep 2017, at 14:54, Stephen Vance<EMAIL_ADDRESS>wrote:
Why didn't they run? Doesn't BOOTSTRAPVERSION=4 ember test work to run the tests under bs4? I don't have the right computer with me to give you exactly what I use. You can use --filter=navbar to reduce the set of tests during development.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/kaliber5/ember-bootstrap/pull/451#issuecomment-331438391, or mute the thread https://github.com/notifications/unsubscribe-auth/AAIu3LdKfIsHgqxnJvR_NWWnthAjKHW7ks5sk63ugaJpZM4PgkIw.

done

On 22 Sep 2017, at 15:56, Stephen Vance<EMAIL_ADDRESS>wrote:
@srvance commented on this pull request.
Nice job. If you can break the test into two tests, I'll approve and merge.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/kaliber5/ember-bootstrap/pull/451#pullrequestreview-64585462, or mute the thread https://github.com/notifications/unsubscribe-auth/AAIu3HmbXEtROS-PW0qsqrIPfBoQBSdGks5sk7yXgaJpZM4PgkIw.

Thank you very much, @basz!


No module named 'inpaint'

Works when generating stereoscopic images, but fails with error below when trying to generate 3D inpainted mesh. Using A1111 1.7.0.
DepthMap v0.4.7 (1e92b5f2)
device: cuda
Loading model(s) ..
Loading model weights from  ./models/leres/res101.pth
initialize network with normal
loading the model from ./models/pix2pix\latest_net_G.pth
Computing output(s) ..
0%|                                                                                | 0/1 [00:00<?, ?it/s]wholeImage being processed in : 896
Adjust factor is: 2.0405783582089554
Selecting patches ...
Target resolution:  (3657, 2845, 3)
Resulting depthmap resolution will be : (1152, 896)
patches to process: 33
processing patch 0 / 32 | [ 35  35 856 856]
processing patch 1 / 32 | [ 35 141 856 856]
processing patch 2 / 32 | [ 35 247 856 856]
processing patch 3 / 32 | [ 96 413 733 733]
processing patch 4 / 32 | [  0   0 712 712]
processing patch 5 / 32 | [  0 106 712 712]
processing patch 6 / 32 | [  0 212 712 712]
processing patch 7 / 32 | [  0 318 712 712]
processing patch 8 / 32 | [  0 423 712 712]
processing patch 9 / 32 | [106   0 712 712]
processing patch 10 / 32 | [242  30 651 651]
processing patch 11 / 32 | [242 136 651 651]
processing patch 12 / 32 | [242 242 651 651]
processing patch 13 / 32 | [312 524 507 507]
processing patch 14 / 32 | [  5 217 487 487]
processing patch 15 / 32 | [  5 323 487 487]
processing patch 16 / 32 | [  5 428 487 487]
processing patch 17 / 32 | [  5 534 487 487]
processing patch 18 / 32 | [217   5 487 487]
processing patch 19 / 32 | [323   5 487 487]
processing patch 20 / 32 | [449  25 446 446]
processing patch 21 / 32 | [449 131 446 446]
processing patch 22 / 32 | [373 479 385 385]
processing patch 23 / 32 | [ 86 192 323 323]
processing patch 24 / 32 | [ 86 827 323 323]
processing patch 25 / 32 | [192 827 323 323]
processing patch 26 / 32 | [297 827 323 323]
processing patch 27 / 32 | [403 827 323 323]
processing patch 28 / 32 | [  0 318 282 282]
processing patch 29 / 32 | [  0 529 282 282]
processing patch 30 / 32 | [  0 635 282 282]
processing patch 31 / 32 | [423   0 282 282]
processing patch 32 / 32 | [529   0 282 282]
100%|█████████████████████████████████████| 1/1 [00:12<00:00, 12.84s/it]
Computing output(s) done.
No module named 'inpaint', some issue with generating inpainted mesh
All done.

I have the same error.

DepthMap v0.4.7 (1e92b5f2)
device: cuda
Loading model(s) ..
Loading model weights from  ./models/leres/res101.pth
initialize network with normal
loading the model from ./models/pix2pix\latest_net_G.pth
Computing output(s) ..
0%|                                                                                | 0/1 [00:00<?, ?it/s]wholeImage being processed in : 1568
Adjust factor is: 1.0
Selecting patches ...
Target resolution:  (1758, 3136, 3)
Resulting depthmap resolution will be : (2595, 4629)
patches to process: 14
processing patch 0 / 13 | [ 992    0 2315 2315]
processing patch 1 / 13 | [1488    0 2315 2315]
processing patch 2 / 13 | [1984    0 2315 2315]
processing patch 3 / 13 | [1228  732 1842 1842]
processing patch 4 / 13 | [1724  732 1842 1842]
processing patch 5 / 13 | [2220  732 1842 1842]
processing patch 6 / 13 | [2716  732 1842 1842]
processing patch 7 / 13 | [<PHONE_NUMBER> 1748]
processing patch 8 / 13 | [<PHONE_NUMBER> 1417]
processing patch 9 / 13 | [ 496    0 1323 1323]
processing patch 10 / 13 | [ 992    0 1323 1323]
processing patch 11 / 13 | [1488    0 1323 1323]
processing patch 12 / 13 | [1984    0 1323 1323]
processing patch 13 / 13 | [2976  496 1323 1323]
100%|██████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:20<00:00, 20.00s/it]
Computing output(s) done.
No module named 'inpaint', some issue with generating inpainted mesh
All done.

@axymeus @Five-Birds I can't reproduce the issue. Could you please provide more info on your environment - do you use standalone more or WebUI mode? What version of WebUI?

As stated, I'm using Automatic 1111 v1.7.0, ran installation through the "install via URL" in extensions tab and reloaded.

@axymeus Did you restart it, too, or just reloaded?

Both. I'm still getting this error today after a complete reboot.

Thanks! Ok.... I've got one lead. Be right back...

I made the fix, hopefully it works now. Please update and test

@axymeus

Fixed on my end.

@axymeus Great! Thank you for helping me


Bump @depyronick/clickhouse-client to 2.0.0

Is there an existing issue that is already proposing this?

[X] I have searched the existing issues

Is your feature request related to a problem? Please describe it

Describe the solution you'd like
  "dependencies": {
    "@depyronick/clickhouse-client": "^2.0.0"
  }

Teachability, documentation, adoption, migration strategy

What is the motivation / use case for changing the behavior?
Use features from 2.0.0

hi @vgorkavenko,
will you create a pull request for that or want me to upgrade?

yep, I will

@depyronick https://github.com/depyronick/nestjs-clickhouse/pull/10

@vgorkavenko thanks
https://github.com/depyronick/nestjs-clickhouse/releases/tag/2.0.0


Add State status constants to spec-go

Hello!
The State definition  defines 4 constants for the status field that are not present in the specs-go
This PR adds them to the state.go file to make it available to runc.
/cc @cyphar @mrunalp @mikebrow
Signed-off-by: Renaud Gaubert<EMAIL_ADDRESS>

LGTM

ping @arnaudwastaken could you have a look at https://github.com/opencontainers/runtime-spec/pull/1046#discussion_r448865229 ?

Sorry looking at it now!

@thaJeztah applied!

@mrunalp @tianon @cyphar PTAL


Add linting

Add linting as documented here: https://docs.astro.build/en/editor-setup/
For now using eslint and stylelint, as Biome does not have the support yet.
Husky pre-commit

https://arc.net/l/quote/lmfuochl

Issues like these are now migrated from Github to Linear for internal use.

Closing because we manage the issues we come up with for now internally via Linear


Alibi embeddings and multi_query_attention do not work together

I am trying to integrate my experimental model architecture, which uses both alibi embeddings and multi_query_attention. In comparing outputs from the vllm-integrated model and my underlying model, I notice that the generations do not match if alibi and multi_query_attention are both turned on. They do match if I turn off multi_query_attention. With a larger model, I actually run into nans, and the generation fails.
I initialize the attention module as follows:
self.attention = PagedAttentionWithALiBi(num_heads, head_dim, scale=scaling, slopes=slopes, num_kv_heads=1)

I have narrowed down the discrepancy to the single_query_cached_kv_attention call. I don't see any test cases for PagedAttentionWithAlibi, so debugging is a little difficult. Any help will be appreciated!

Closing this issue as stale as there has been no discussion in the past 3 months.
If you are still experiencing the issue you describe, feel free to re-open this issue.


evaluate returns only null or Promise {  }

I have tried this many different ways and I'm always getting null or Promise {  }
var element = sitepage.evaluate(function(x, y) {
 	return document.elementFromPoint(x, y).outerHTML;
}, x, y);

It should return HTML element from given location.
Here is where I'm trying to achive this:
var phantom = require('phantom');
var sitepage;
var url = "http://amazon.com";

phantom.create().then(function(ph){
	return ph.createPage();
}).then(function(page){
	var pageWidth = parseInt("1920px", 10);
	var pageHeight = parseInt(pageWidth * 3/4, 10);
	sitepage = page;
	sitepage.property('viewportSize', { width: pageWidth, height: pageHeight });
	return sitepage.open(req.query.url);
}).then(function(status){
	if (status == "success") {
		var x = 888;
		var y = 777;

		var element = sitepage.evaluate(function(x, y) {
		 	return document.elementFromPoint(x, y).outerHTML;
		}, x, y);

		return element;
	}

	return '404';
}).then(function(content){
	console.log(content);

	page.close();
	ph.exit();
});

Here is the example from README:
page.evaluate(function() {
    return document.getElementById('foo').innerHTML;
}).then(function(html){
    console.log(html);
});

Even if I try to return document.title or somehting else, still getting null or Pormise {  }. I'm sorry if this doesn't belong to here, but have spent many many hours with this one and still not working :/
I have also tried that example found in README, but still no success. Or maybe I'm not using it right..
Any ideas, or is there anyone who got this working?
Environment

OS: OSX
Node version: v4.4.4
package version: 2.1.3

This is the right place to ask questions like this. I'll check it tonight when I get home.

Okay, I solved the original issue and it was caused by this line sitepage.property('settings', {loadImages: false}); (marked by * in issue). That line didn't cause error, but by removing that line I accidentally noticed that my code started working. Not sure if this should be a new issue?

You have to use a different method to set settings. See readme.
Sent via my iPhone

On Jun 5, 2016, at 10:24 AM, Juho Lehtinen<EMAIL_ADDRESS>wrote:
Okay, I solved the original issue and it was caused by this line sitepage.property('settings', {loadImages: false}); (marked by * in issue). That line didn't cause error, but by removing that line I accidentally noticed that my code started working. Not sure if this should be a new issue?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

Ok now that I am home. You need to use page.setting(key)


add symlinks for capistrano back in

I must listen to @h-lame ...

Partly my fault - I should have asked about these more publicly, then others might have chipped in and we wouldn't have had to do this dance.  I guess I just didn't want to reveal I don't know how our DO deployment process works.


python37Packages.cliff: 3.6.0 -> 3.7.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python37Packages.cliff is: "Command Line Interface Formulation Framework"
meta.homepage for python37Packages.cliff is: "https://docs.openstack.org/cliff/latest/"
meta.changelog for python37Packages.cliff is: ""
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS

Warning: a test defined in passthru.tests did not pass

0 of 0 passed binary check by having a zero exit code.

0 of 0 passed binary check by having the new version present in output.

found 3.7.0 with grep in /nix/store/kcvz9dirx2igz7nvyw6nzdli7b0y5d5j-python3.7-cliff-3.7.0

directory tree listing: https://gist.github.com/ceab0847bf3c42c26e0284efe84abe90

du listing: https://gist.github.com/6c95f911b27673b9e2249af5767da359

Rebuild report (if merged into master) (click to expand)

21 total rebuild path(s)

6 package rebuild(s)

6 x86_64-linux rebuild(s)
3 i686-linux rebuild(s)
6 x86_64-darwin rebuild(s)
6 aarch64-linux rebuild(s)

First fifty rebuilds by attrpath
python37Packages.cliff
python37Packages.optuna
python38Packages.cliff
python38Packages.optuna
python39Packages.cliff
python39Packages.optuna

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/kcvz9dirx2igz7nvyw6nzdli7b0y5d5j-python3.7-cliff-3.7.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python37Packages.cliff https://github.com/r-ryantm/nixpkgs/archive/1d62ff78cdcd5a941f6195337c46045404dbce4f.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/kcvz9dirx2igz7nvyw6nzdli7b0y5d5j-python3.7-cliff-3.7.0
ls -la /nix/store/kcvz9dirx2igz7nvyw6nzdli7b0y5d5j-python3.7-cliff-3.7.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

4 packages marked as broken and skipped:

python37Packages.optuna
python38Packages.optuna
python39Packages.cliff
python39Packages.optuna

2 packages built:

python37Packages.cliff
python38Packages.cliff

Maintainer pings
cc @costrouc for testing.

Result of nixpkgs-review pr 113514 at 1d62ff78 run on aarch64-linux 1

4 packages marked as broken and skipped:

python37Packages.optuna
python38Packages.optuna
python39Packages.cliff
python39Packages.optuna

2 packages built:

python37Packages.cliff
python38Packages.cliff


bug: NPE on upgrade

Background
I'm updating DGS libraries from 4.10.2 to 5.5.3 due to newline(\n) handling, then running a @SpringBootTest integration test which uses DgsQueryExecutor directly to fire queries to the running test server.
Expected behavior
Tests pass. Without issue.
Actual behavior
The bug https://github.com/Netflix/dgs-framework/issues/1105 gets reproduced exactly.
TLDR: Custom headers throw NPE due to requestData.webRequest being null in the context.
Steps to reproduce
Use The following relevant lib versions:

id 'org.springframework.boot' version '2.7.0'
implementation(platform('com.netflix.graphql.dgs:graphql-dgs-platform-dependencies:5.5.3'))
implementation('com.netflix.graphql.dgs:graphql-dgs-spring-boot-starter:5.5.3')
implementation('com.netflix.graphql.dgs:graphql-dgs-extended-scalars:5.5.3')

_Note:
THis is fixed in https://github.com/Netflix/dgs-framework/pull/1421 but is not contained in 5.5.3
Workaround:
Use a MockHttpServletRequest to pass the headers and wrap the DgsQueryExecutor bean in the text context with a custom implementation of DgsQueryExecutor overriding all methods that pass headers.

Hi @GellertH - We decided to add new features and fixes in the Spring Boot 3 compatible versions of DGS , i.e >6.x. Unless there are major security fixes, or other critical fixes, we are not backporting changes to reduce our maintenance overhead. Please switch to the latest 7.x version if possible.


ENH: allow the contingency wrapper to eat exceptions

Description

Add an option to contingency_wrapper to not automatically re-raise if the except plan returns a value rather than raising its own exception.
We have to do this with a flag like this for back-compatibility reasons
Motivation and Context

We (I) made a design mistake with contingency_wrapper because there is no way to actually suppress the exception.  The reason we wrote this wrapper (and finalize wrapper) is to keep users from having to understand the extra set of constraints around the cancel and generator exit exceptions so this omission defeats the point.
How Has This Been Tested?

Needs tests!

This would be useful for us at DLS, can I pick it up and write tests etc?

@DominicOram yes please.

Continued in https://github.com/bluesky/bluesky/pull/1599


Automating Shadow DOM elements - use dom step and JavaScript to interact

Hi @kensoh,
How do I automate an element that is inside Shadow DOM? Using Xpath and r.click() is not working.
Regards,
Renato.

This site has an example with google downloads. Hope it is helpful. However, it is based on selenium
https://www.seleniumeasy.com/selenium-tutorials/accessing-shadow-dom-elements-with-webdriver

Hi Renato, this will require learning JavaScript to know what standard JS functions to use for selecting dropdown box or clicking a button. I'm afraid googling to learn more is the only path, same as above if you use Selenium's method.
Need to send the raw JavaScript to do the automation on the shadow dom.
You can read TagUI's codebase, of course, but that will probably take 10X more time than googling the JavaScript code directly. Because a lot is done in TagUI's backend to perform a simple action, to make it work most reliably.

Hi @kensoh found thee solution using JS function calls. Code is longer than expected
r.dom("root1 = document.querySelector('body > app-root > div > app-queue > div > app-workqueue-tabs > div > div.row.proton-container__workqueue-cards > div > app-workqueue-cards > div > div:nth-child(4) > scc-filters').shadowRoot")
r.dom("return root1.querySelector('div > div.container > div:nth-child(9) > div.label').click()")

r.dom("root1 = document.querySelector('body > app-root > div > app-queue > div > app-workqueue-tabs > div > div.row.proton-container__workqueue-cards > div > app-workqueue-cards > div >div:nth-child(4) > scc-filters').shadowRoot")
r.dom("root2 = root1.querySelector('div > div.container > div:nth-child(9) >div.dropdown-container.ng-star-inserted > div.dropdown-content > div > scc-filter-item').shadowRoot")
r.dom("root3 = root2.querySelector('div:nth-child(23) > scc-checkbox-filter').shadowRoot")
r.dom("return root3.querySelector('div.search > -textbox > div > input').value("01")")

Oh great! Yes, it is a pain to navigate shadow dom. Thanks for sharing back your solution.
In this case you have to navigate 3 layers down before doing the actual selection.

hi @kensoh,
With the above solution i'm able to send values. When I manually enter values the filter shows that specific values. When it is sent via code the filter is not working.
Any suggestions on this

A) Do you mean this works -
r.dom("return root3.querySelector('div.search > -textbox > div > input').value=01")

B) But this fails?
r.dom("return root3.querySelector('div.search > -textbox > div > input').value="+co)

The code in B looks correct, so I don't know why A works but B fails.
But you have typo in the variable RC rc=['01','02]. Check carefully your code again.

A) Do you mean this works -
r.dom("return root3.querySelector('div.search > -textbox > div > input').value=01")

B) But this fails?
r.dom("return root3.querySelector('div.search > -textbox > div > input').value="+co)

The code in B looks correct, so I don't know why A works but B fails.
But you have typo in the variable RC rc=['01','02]. Check carefully your code again.

Both dont work for the auto filtering part. When i enter the values manually only that value reflects but with the code all the values in the system reflect.

I see, you can try out more using the Chrome console how to do this or try using visual automation method.

I see, you can try out more using the Chrome console how to do this or try using visual automation method.

Hi @kensoh, the item that i'm trying to send data to is not a image icon. How do i use visual automation to click on it.

Yes visual automation can be anything on the screen, for eg some text box can also be captured.
See this - https://github.com/tebelorg/RPA-Python#visual-automationsee_no_evil
And this - https://github.com/aimakerspace/TagUI-Bricks/tree/master/MS-Word-Letter
Use r.keyboard() to send keystrokes, it's on the GitHub page -
https://github.com/tebelorg/RPA-Python#keyboard-automationmusical_keyboard

@kensoh i tried and it did not work. How could i get this code to work. Which is written in selenium java. I tried every thing and it did not work.
case "Reason Code":
        filterElement = getSccFilterShadowRootElement()
                .findElement(By.cssSelector("div.container div[class*=ng-star-inserted]:nth-of-type(9)"));

If you are using dom(), in JavaScript, it will be root3.querySelector("div.container div[class*=ng-star-inserted]:nth-of-type(9)"), assuming root3 is the shadow dom you derived here - https://github.com/tebelorg/RPA-Python/issues/295#issuecomment-916303989
r.dom("return root3.querySelector('div.search > -textbox > div > input').value="+co) should be r.dom("return root3.querySelector('div.search > -textbox > div > input').value= '"+co + "'") if co variable is a string.
Otherwise you can use visual automation to control Chrome, using here - https://github.com/tebelorg/RPA-Python/issues/295#issuecomment-918164969


Readability redundancy improvements

refactor: split extension class into 2 files
Split extension class into 2 files, one for just double the other for just float (makes it easier to tell that they are nearly identical).
test: updated tests to be less redundant
Updated tests to make use of XUnit MemberData so there is less redundancy in test code while making multiple use of the test scenarios. There is a set of test scenarios for:

Equals (used by EqualsTo, IsGreaterThanOrEquals, and IsLessThanOrEquals)
Not Equals (used by EqualsTo)
Zero (used by IsZero)
Comparison (Used by IsGreaterThan, IsGreaterThanOrEquals, IsLessThan, and IsLessThanOrEquals)

@Commit-Lint review

I'm not sure what commit-lint's issue is with the description on this PR.

I'm not sure what commit-lint's issue is with the description on this PR.

Commit messages must follow the conventional commits spec in order to deploy the packge with the correct version. I see that you edit the PR description accordingly. However the PR description does not matter. It is commit message that matters.

Since all I'm really doing here is changing tests, what do you think about adding netstandard1.0 so this can be used by more than just net5.0? I'm happy to add that in this or a different PR. That would be a good reason to push a new version to nuget.

apparently there is still an issue with these commit messages. I've read https://www.conventionalcommits.org/en/v1.0.0/ and the commit messages I have on the 2 commits should fit the convention. I'd appreciate any help you can give for why they don't. I would be certainly be nice for this lint tool to give feedback about what's wrong.


coordinate efforts on Go tooling for protolock

Hey - it was brought to my attention that there may be some efforts to port protoman backward compatibility tooling to Go, and thought I would share a tool the community has helped improved significantly since my initial release: protolock
There have been a number of community-driven PRs and usage from within orgs including Fanatics, Lyft (internal and Envoy), Salesforce, Google (Istio), and more. If Spotify were interested in adopting the tool as well, I'm sure we can plan to incorporate features protoman supports that protolock does not yet. We recently added support for plugins (a la protoc-style executables), which can be written in any language and extend the base rules baked into protolock.
If you have already made progress elsewhere, ignore this and carry on :) -- just figured I'd share another option with hopes of bringing more skilled contributors together.
Thanks!

Hey @nilslice, first of all thanks for reaching out. We are really interested in improving the tooling for proto management and validation. This is the first time I hear about protolock and it looks cool :)
Our public protoman in it's current form has been put on hold. We have however rewritten and simplified it to just a CLI in Golang which just does fetch in it's current form. The registry is a git repo with all our protos, protoman is using the github API to fetch the specified package and it's depenedencies. We are using prototool by lyft as a build step to do validation in our shared proto repo when users publish new changes.
We would like to incorporate some of our own lint checks and it would be awesome to put everything into one tool that can be extended. Since the current version of the protos are stored in git we are planning to rely on git comparison between master and the current pr/branch to calculate the state changes instead of having to commit an additional lock file. Do you think that could be something that could be incorporated into protolock as a new/secondary way of comparing the state?
I will bring this up with my team and also see if we can upload our current iteration of protoman to github to give some more context on how we currently use our tool.
Thanks again for reaching out!

Thanks for the reply, and I apologize for my delayed response!

Since the current version of the protos are stored in git we are planning to rely on git comparison between master and the current pr/branch to calculate the state changes instead of having to commit an additional lock file.

I considered using Git as a tool to track and source proto versions as well, but have concluded that the lock file is more efficient and less restrictive. Otherwise, using git is a strict requirement (some users have other VCS), and using protoc is a requirement, since generating the descriptor set must be done before the comparison. Using protoc is not as big of a deal, as we are working with protobufs after all :)
Avoiding the use of a lock file is completely understandable, but I've found it is actually quite nice to have a readable, tracked reference to the state of your protos.

Do you think that could be something that could be incorporated into protolock as a new/secondary way of comparing the state?

Personally, I find the use of the descriptor set a bit more cumbersome to work with when comparing the states. I'm looking forward to seeing any real code that does the comparisons, and code to provide extensible plugins for adding custom checks/rules - to my knowledge, this has not started or reached a state where it could be released. Although I'd like to say it could be supported in protolock, I have no plans to start that work and add git and protoc as dependencies. I think the right path is a standalone tool that covers a smaller set of responsibilities.
I would be really interested to see your current iteration, but understand some things need to stay internal :)


chore: Release v4.14.0 notes and site improvements

:tada: This PR is included in version 4.14.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Add universal redirect example

Would be nice to have a deployment on now to see how it behaves 😊

@sedubois great idea, done 🙌

What about https://github.com/matthewmueller/next-redirect by @matthewmueller?

I think next-redirect is easier to use than next-universal-redirect, is just a function you call in getInitialProps, you don't need a HOC and create a Map instance, just call a simple function 🤔

@sergiodxa it serves a different purpose. This one is for lists of redirects from non-existent pages to existent (well, if you screw up non-existent) pages. Next-redirect is for a more for redirecting when stuff happens. Also, next-redirect is not universal but only server side.

next-redirect is not universal but only server side.

actually it works on both: https://github.com/matthewmueller/next-redirect/blob/master/index.js#L16

@matthewmueller ah, right, I meant it doesn't use next/router 😄


Customizing training data for fine-tuning the model

Hello,
I'm trying to use my own dataset to fine-tune DNABERT-5/6.
I have several questions about this.

All input sequence length should be the same? Or input sequences can be different length?
Does label should be 1/0? Basically, does classification class can be more than two classes?

Any feedback would be appreciated.
Thank you!

Excuse me, do you understand these two questions？

yes, i was wondering whether the labels could be 1 to 10 rather than binary classify task


Add support for importing CSV datasets

Adds support for importing datasets stored as CSV files.
Example usage:
import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("cifar10")

dataset.export(
    labels_path="/tmp/cifar10.csv",
    dataset_type=fo.types.CSVDataset,
    fields={
        "filepath": "filepath",
        "tags": "tags",
        "ground_truth.label": "ground_truth",
    },
    abs_paths=True,
)

dataset2 = fo.Dataset.from_dir(
    labels_path="/tmp/cifar10.csv",
    dataset_type=fo.types.CSVDataset,
    fields={
        "filepath": None,  # load as strings
        "tags": lambda v: v.strip("").split(","),
        "ground_truth": lambda v: fo.Classification(label=v) if v else None,
    },
)

assert set(dataset.values("filepath")) == set(dataset2.values("filepath"))
assert dataset.count_sample_tags() == dataset2.count_sample_tags()
assert dataset.count_values("ground_truth.label") == dataset2.count_values("ground_truth.label")

Notes on v1 of this implementation:
Potential TODO

I don't love the class Parsers that I ended up with. Open to suggestions for improvement. The good thing about it is it's not strict so any custom function or lambda can be passed as a parser for a field.
Split CSV read so we can do some row parsing in parallel?
Any corner cases I may have missed?
Add more common Parsers to add? (May get much more complicated if we try for detections because that is nested lists

What changes are proposed in this pull request?
Create CSVDatasetImporter complementary to CSVDatasetExporter.
Most unique aspect ( the field parsing ) should be called out, and it's described in the docstring which I've pasted here.
fields (None): An optional parameter that specifies fields to read and parse
    from the CSV file. Can be any of the following:
    -   `None`. Will use all fields in the CSV and parse with default
        string parser.
    -   `List[str]` which are the field names to be read from
        the CSV. All data will be parsed with default string parser.
    -   `Dict[str, Optional[fiftyone.utils.csv.Parsers]]` where the key
        specifies the field name to be read and the value specifies the
        parser to be used, e.g., `StringParser` or `IntParser`. If `None`,
        default string parser will be used. `ListParser` can be uniquely
        used by passing in sub parser type to use for each data field.
        For example:
        {
            "float_field": Parsers.FloatParser,
            "list_field": Parsers.ListParser(Parsers.StringParser),
            "classification_field": Parsers.ClassificationParser,
            "string_field": None # uses DefaultParser
        }


Misunderstading the error

Heya! I have a question. In the table of Operations there is an operation called 'help' which, conform the description should print the logo screen and the list of operations available when the command 'node main.js help' is typed. But, then in the Erros section there is a given  line of command
'node main.js help' which should print an error because the 'help' command doesn't exists, and to use
display to see all the commands. And my question is: does the command 'help' exists or not? How should it print the commands available(bc that's what is saying in the table) when  it is put in the Error section?

The display command does not exist, the command is help.
On Mon, Feb 24, 2020 at 11:29 PM daniciucBianca<EMAIL_ADDRESS>wrote:

Heya! I have a question. In the table of Operations there is an operation
called 'help' which, conform the description should print the logo screen
and the list of operations available when the command 'node main.js help'
is typed. But, then in the Erros section there is a given line of command
'node main.js help' which should print an error because the 'help' command
doesn't exists, and to use
display to see all the commands. And my question is: does the command
'help' exists or not? How should it print the commands available(bc that's
what is saying in the table) when it is put in the Error section?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/UPB-FILS/alf/issues/1?email_source=notifications&email_token=AAVU5IMY5XPSKTSJDJ4SBALREQ325A5CNFSM4K2S4FD2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IP35SIA,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAVU5IPURX5EVZB4YDWO7EDREQ325ANCNFSM4K2S4FDQ
.

--
Alexandru RADOVICI
Universitatea "Politehnica" din Bucuresti
e-mail<EMAIL_ADDRESS>telefon:<PHONE_NUMBER>
www.ipworkshop.ro


TextFormField/TextField's style doesn't inherit color

Steps to Reproduce

Run flutter create bar
Replace main function's content on

void main() => runApp(MaterialApp(
      home: Material(
        child: Center(
          child: Builder(
            builder: (BuildContext context) {
              return TextFormField(
                style: TextStyle(fontSize: 24.0),
              );
            },
          ),
        ),
      ),
    ));

Start app
Type something in input

AR: Color is white
ER: Color from material theme (black)
Logs

Analyzing bar...
No issues found! (ran in 1.1s)

[✓] Flutter (Channel beta, v0.9.4, on Mac OS X 10.14.1 18B75, locale en-RU)
    • Flutter version 0.9.4 at /www/dart/flutter
    • Framework revision f37c235c32 (6 weeks ago), 2018-09-25 17:45:40 -0400
    • Engine revision 74625aed32
    • Dart version 2.1.0-dev.5.0.flutter-a2eb050044

[✗] Android toolchain - develop for Android devices
    ✗ ANDROID_HOME = /usr/local/opt/android-sdk
      but Android SDK not found at this location.

[✓] iOS toolchain - develop for iOS devices (Xcode 10.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 10.1, Build version 10B61
    • ios-deploy 2.0.0
    • CocoaPods version 1.5.3

[✗] Android Studio (not installed)
    • Android Studio not found; download from https://developer.android.com/studio/index.html
      (or visit https://flutter.io/setup/#android-setup for detailed instructions).

[!] IntelliJ IDEA Ultimate Edition (version 2018.3 EAP)
    • IntelliJ at //Applications/JetBrains Toolbox/IntelliJ IDEA Ultimate.app
    ✗ Flutter plugin not installed; this adds Flutter specific functionality.
    ✗ Dart plugin not installed; this adds Dart specific functionality.
    • For information about installing plugins, see
      https://flutter.io/intellij-setup/#installing-the-plugins

[✓] Connected devices (1 available)
    • iPhone XR • 28E7063A-3409-43F9-AB68-CDD9218CF571 • ios • iOS 12.1 (simulator)

! Doctor found issues in 3 categories.

I have investigated the problem and found that if you pass custom style without color then the color will be equal to null when building TextSpan in EditableText.
As a solution - use DefaultTextStyle.of to fulfill null values in style object
diff --git a/packages/flutter/lib/src/widgets/editable_text.dart b/packages/flutter/lib/src/widgets/editable_text.dart
index 3c79cc6ac..0389fe235 100644
--- a/packages/flutter/lib/src/widgets/editable_text.dart
+++ b/packages/flutter/lib/src/widgets/editable_text.dart
@@ -21,6 +21,7 @@ import 'scroll_controller.dart';
 import 'scroll_physics.dart';
 import 'scrollable.dart';
 import 'text_selection.dart';
+import 'text.dart';

 export 'package:flutter/services.dart' show TextEditingValue, TextSelection, TextInputType;
 export 'package:flutter/rendering.dart' show SelectionChangedCause;
@@ -917,13 +918,15 @@ class EditableTextState extends State<EditableText> with AutomaticKeepAliveClien
   /// By default makes text in composing range appear as underlined.
   /// Descendants can override this method to customize appearance of text.
   TextSpan buildTextSpan() {
+    final style = DefaultTextStyle.of(context).style.merge(widget.style);
+
     if (!widget.obscureText && _value.composing.isValid) {
-      final TextStyle composingStyle = widget.style.merge(
+      final TextStyle composingStyle = style.merge(
         const TextStyle(decoration: TextDecoration.underline),
       );

       return TextSpan(
-        style: widget.style,
+        style: style,
         children: <TextSpan>[
           TextSpan(text: _value.composing.textBefore(_value.text)),
           TextSpan(
@@ -942,7 +945,8 @@ class EditableTextState extends State<EditableText> with AutomaticKeepAliveClien
       if (o != null && o >= 0 && o < text.length)
         text = text.replaceRange(o, o + 1, _value.text.substring(o, o + 1));
     }
-    return TextSpan(style: widget.style, text: text);
+
+    return TextSpan(style: style, text: text);
   }
 }

Fixed!  It should inherit from the theme now.


⚠ Samsung has degraded performance

In e8ec538, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2953 ms

Resolved: Samsung performance has improved in e56ecb3.


Upgrade ULTRA's setup.py

Upgrade ULTRA's setup.py to use a SMARTS version that does not require X11 and can be used for the release of the next version.
TODO: Upgrade the SMARTS dependency to 0.4.19 once it is released (due to an inconsistency in SMARTS 0.4.18).
Will require a review from Jenish and Adai before merging.

@Adaickalavan, do you know what could be causing the problem in ULTRA's light Tune test? It seems it occurs when we are trying to instantiate the RemoveAgentBuffer for SMARTS' AgentManager (https://github.com/huawei-noah/SMARTS/pull/1027/checks?check_run_id=3162260242).

Hi @richardliaw

The test case, which makes use of the source code, fails.

We get an error message:
"ValueError: signal only works in main thread"

Raw log file of the failure can be obtained from here.

Do you know of any methods to overcome this?

@Adaickalavan it looks like signal.signal is not supported in Ray. Can you try instead using an exit handler (atexit)?


Parallel loop failing with java.lang.NoSuchMethodError: java.nio.ByteBuffer.limit(I)Ljava/nio/ByteBuffer;

Hi, trying to run a parallel loop using Digdag latest version, and running into a cryptical Java error.
Here's the error trace:
io.digdag.core.agent.OperatorManager: Task failed with unexpected error: java.nio.ByteBuffer.limit(I)Ljava/nio/ByteBuffer;
java.lang.NoSuchMethodError: java.nio.ByteBuffer.limit(I)Ljava/nio/ByteBuffer;
	at io.digdag.core.database.AESGCMSecretCrypto.decryptSecret(AESGCMSecretCrypto.java:173)
	at io.digdag.core.database.DatabaseSecretStore.getSecret(DatabaseSecretStore.java:49)
	at io.digdag.core.agent.GrantedPrivilegedVariables.lambda$privilegedSecretProvider$0(GrantedPrivilegedVariables.java:25)
	at io.digdag.spi.SecretProvider.getSecret(SecretProvider.java:20)
	at io.digdag.util.UserSecretTemplate.lambda$format$0(UserSecretTemplate.java:55)
	at io.digdag.util.UserSecretTemplate.replaceAll(UserSecretTemplate.java:65)
	at io.digdag.util.UserSecretTemplate.format(UserSecretTemplate.java:55)
	at io.digdag.core.agent.GrantedPrivilegedVariables.lambda$buildAccessor$1(GrantedPrivilegedVariables.java:62)
	at io.digdag.core.agent.GrantedPrivilegedVariables.get(GrantedPrivilegedVariables.java:81)
	at io.digdag.util.CommandOperators.collectEnvironmentVariables(CommandOperators.java:18)
	at io.digdag.standards.operator.PyOperatorFactory$PyOperator.runCommand(PyOperatorFactory.java:296)
	at io.digdag.standards.operator.PyOperatorFactory$PyOperator.runCode(PyOperatorFactory.java:159)
	at io.digdag.standards.operator.PyOperatorFactory$PyOperator.runTask(PyOperatorFactory.java:115)
	at io.digdag.util.BaseOperator.run(BaseOperator.java:35)
	at io.digdag.core.agent.OperatorManager.callExecutor(OperatorManager.java:365)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invokeMain(DigdagTimedMethodInterceptor.java:58)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invoke(DigdagTimedMethodInterceptor.java:31)
	at io.digdag.core.agent.OperatorManager.runWithWorkspace(OperatorManager.java:298)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invokeMain(DigdagTimedMethodInterceptor.java:58)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invoke(DigdagTimedMethodInterceptor.java:31)
	at io.digdag.core.agent.OperatorManager.lambda$runWithHeartbeat$2(OperatorManager.java:151)
	at io.digdag.core.agent.ExtractArchiveWorkspaceManager.withExtractedArchive(ExtractArchiveWorkspaceManager.java:77)
	at io.digdag.core.agent.OperatorManager.runWithHeartbeat(OperatorManager.java:149)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invokeMain(DigdagTimedMethodInterceptor.java:58)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invoke(DigdagTimedMethodInterceptor.java:31)
	at io.digdag.core.agent.OperatorManager.run(OperatorManager.java:132)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invokeMain(DigdagTimedMethodInterceptor.java:58)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invoke(DigdagTimedMethodInterceptor.java:31)
	at io.digdag.core.agent.MultiThreadAgent.lambda$run$0(MultiThreadAgent.java:132)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Any help would be very appreciated.
Thank you!

Hello, @FredericoCoelhoNunes
Could you provide us about your environment?

Digdag version: latest means 10.0.1?
Reproducable dig file
Java version
OS Version

Hi @hiroyuki-sato , sorry for the delay.
Here is the info:

Digdag version: obtained from https://dl.digdag.io/digdag-latest, although the output of diagdag --version is 0.10.1?
Java version: output of java -version is:

openjdk version "1.8.0_292"
OpenJDK Runtime Environment (build 1.8.0_292-8u292-b10-0+deb9u1-b10)
OpenJDK 64-Bit Server VM (build 25.292-b10, mixed mode)

OS version: Output of lsb_release -da:

No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 9.13 (stretch)
Release:        9.13
Codename:       stretch

As for a reproducible example:

dig file:

+my_test:
  _parallel: true
  for_each>:
    schema: [
      {
        file:     "MY-FILE"
      }
    ]
  _do:
    # Executor
    py>: test_module.test_print
    _env:
      MY_SECRET:         ${secret:MY_SECRET}

test_module.py file:

import digdag

def test_print():
    print(f'{schema["file"]}')

Setting up:

digdag push --project . test_project -e localhost:9090
digdag secrets --project test_project --set MY_SECRET=12345 -e localhost:9090

A few final important notes:

the problem seems to be in the secrets
the problem was solved by downgrading to Digdag version 0.9.42

Thanks!!

Hello, @FredericoCoelhoNunes
Did you setup secret-encryption-key ?

Hi @hiroyuki-sato , forgot to specify that in my minimum reproducible example, but yes I did.
Technically I also have a Postgres database running with docker-compose and I set up Digdag in the following way:
echo "
server.bind=<IP_ADDRESS>
server.port=9090
server.admin.bind=<IP_ADDRESS>
server.admin.port=9091
database.maximumPoolSize=10
database.type=postgresql
database.user=${DIGDAG_DB_USER}
database.password=${DIGDAG_DB_PASSWORD}
database.host=${DIGDAG_DB_HOST}
database.port=${DIGDAG_DB_PORT}
database.database=${DIGDAG_DB_NAME}
digdag.secret-encryption-key=<MY KEY>
" > digdag_server.properties

digdag server \
  --task-log ./digdag/sessions/ \
  --max-task-threads 5 \
  --config ./digdag_server.properties

Hello, @FredericoCoelhoNunes
Thank you for your report. I could reproduce it.
https://github.com/hiroyuki-sato/digdag-support/tree/master/20210720_core_1613
I'll investigate it. Please use 0.9.x until fix this issue.

No worries @hiroyuki-sato , thanks for verifying. Keep up the great work!

@fernandomrtnz Thanks for reporting this issue! And @hiroyuki-sato Thanks for your support as always 😉
I also reproduced this issue with 0.10.1 while didn't reproduce with 0.10.0. Let me look into it.

This exception doesn't happen with 0.10.1 built from source code. Looks like a packaging issue?

Hmm. I checked the diff (especially around io/digdag/core/database/AESGCMSecretCrypto.class) between https://github.com/treasure-data/digdag/releases/download/v0.10.1/digdag-0.10.1.jar and https://github.com/treasure-data/digdag/releases/download/v0.10.0/digdag-0.10.0.jar, but didn't find suspicious diff...
@szyn Do you still have digdag-0.10.1.jar which you built and used for the release on you laptop or somewhere? If so, could you try to reproduce this issue with it when you have time?

I guess 0.10.1 may be built with Java11.

May be helpful information.
https://github.com/eclipse/jetty.project/issues/3244

Hmm. I compiled v0.10.1 source code using Java 11 and tested it, but I didn't reproduce this issue. This issue happened only with the released jar on my side.
We've specified targetCompatibility = 1.8 https://github.com/treasure-data/digdag/blob/v0.10.1/build.gradle#L56, so generated class files should be compatible with Java 8. I also checked io/digdag/core/database/AESGCMSecretCrypto.class as mentioned earlier and the target major version is Java 8.
Classfile xxxxx/0.10.1/io/digdag/core/database/AESGCMSecretCrypto.class
  Last modified Jun 4, 2021; size 6116 bytes
  MD5 checksum ac3b4f562b60b087bdfdb5c5c953ec5b
  Compiled from "AESGCMSecretCrypto.java"
public class io.digdag.core.database.AESGCMSecretCrypto implements io.digdag.core.crypto.SecretCrypto
  minor version: 0
  major version: 52

Hmm. I compiled v0.10.1 source code using Java 11 and tested it, but I didn't reproduce this issue. This issue happened only with the released jar on my side.

Oops. My test was wrong. I thought I had run the executable using Java 8, but I actually ran it with Java 11 that was the same version with the build. I reproduced this issue when using Java 8. Sorry for the confusion.

Maybe we'd better explicitly add --release 8 option to javac like this
diff --git a/build.gradle b/build.gradle
index 240042c7a8..6167dc0c04 100644
--- a/build.gradle
+++ b/build.gradle
@@ -55,7 +55,10 @@ subprojects {
     sourceCompatibility = 1.8
     targetCompatibility = 1.8

-    compileJava.options.encoding = 'UTF-8'
+    compileJava {
+        options.encoding = 'UTF-8'
+        options.compilerArgs.addAll(['--release', '8'])
+    }
     compileTestJava.options.encoding = 'UTF-8'

     dependencies {

With this option, I confirmed this issue isn't reproduced even with the combination of Java 11 for build and Java 8 for execution.
But, of course, the easiest fix is to build with Java 8  and release as 0.10.2 😉

Thank you for reporting this issue, I also could reproduce this by using Java8.
As @yoyama pointed out, this is the reason why I build digdag with Java 11.

Maybe we'd better explicitly add --release 8 option to javac like this

Thanks, that's good to know, and agree with you.

But, of course, the easiest fix is to build with Java 8 and release as 0.10.2 😉

I think so, so let me build with Java 8 and release it as 0.10.2 later.

@komamitsu @yoyama @szyn Thank you for your reply.
JFYI. @dmikurube 's comment on Twitter
Rough translate in English: It is necessary to build with Java 8 if Digdag supports Java 8.
In my opinion, dropping Java 8 supports is another idea if many contributors use Java 11 mainly.

Hello, @FredericoCoelhoNunes
Digdag 0.10.2 has been released.
Could you try it?
You can update using digdag selfupdate 0.10.2.

Hi @hiroyuki-sato , unfortunately I won't be able to test it in the near future as I have moved to a different project. Thanks!

Hi, @FredericoCoelhoNunes Thank you for your comment.
@szyn I confirmed this issue has been fixed in Digdag 0.10.2. Thanks!

@hiroyuki-sato I'd appreciate your confirmation!
Let me close this issue.


Problems with Cron after updating to Drupal Core 7.41

"After other bugs have been fixed, only edgesense keeps making problems. Issue
is an incomplete implementation of edgesense_cron, which is using hook_cron()
directly. It should use Queue API instead, as documented on the code.
Changing the PHP memory limit to 1024 Mb on the server overcomes the issue,
but a new issue for this performance killer of edgesense_cron() should
nevertheless be open." http://edgeryders.eu/en/comment/21381#comment-21381

Superseded by #101 and #102


Do you train the LLM in the first stage?

In section 4.1, We train the projector and LM backbone, and freeze the vision encoder at both stages.
Do you train the LLM in the first stage?

Yes. The trainabilities of each module are the same for both stages (as summarized in Table 2).
Thanks for pointing this out! We'll state it clearer in the paper.

Maybe it's a little unfair. The LLaVA-based models you compare did not train LLM in first stage. Have there been further ablation experiments? My concern is the training parameters.

Hi @LinB203 ,
Thanks for your question. This is indeed a very necessary ablation that we've missed.

Models we should compare with
Among the three 3B-scale models we compared in our paper, TinyGPT-V adopts a 4-stage training pipeline, while MobileVLM and LLaVA-Phi both adopt LLaVA-like training pipelines. Hence, to address your concern, we only compare the LLaVA-like models.

Experiment Setting
To have a fair comparison, we only train the projector at PT and adjust the peak LR to 1e-3 at PT. The other settings follow the settings of ALLaVA-Longer. The resulted model is named ALLaVA-only_proj@PT.

Validity of our ALLaVA-4V dataset
The validity of our dataset is manifested in two fashions. Results are detailed in the table below.

Comparison with MobileVLM and LLaVA-Phi
ALLaVA-only_proj@PT  performs significantly better than MobileVLM and LLaVA-Phi, but in general performs slightly worse than ALLaVA-Longer due to the decrease of trainable parameters at PT.

Comparison with models with larger scale
ALLaVA-only_proj@PT also performs on par with some 7B models on several benchmarks.

Model
MMB
SEED (v1, img)
MM-Vet
MME
TextVQA
GQA

Qwen-VL-Chat
60.6
65.4
-
1487.5
61.5
57.5

LLaVA-v1.5-7B
64.3
-
31.1
1510.7
58.2
62.0

MobileVLM
59.6
-
-
1288.9
47.5
-

LLaVA-Phi
59.8
-
28.9
1335.1
48.6
-

ALLaVA-Longer
64.6
65.6
35.5
1564.6
50.3
50.0

ALLaVA-only_proj@PT
65.3
64.4
33.7
1557.1
50.3
49.1

Hope this ablation addresses your concern!
Best,
Guiming

Great work! I will take your data!

Thanks! We are releasing the LAION images to hf in a few days. Stay tuned!

We have updated the laion images as well as the download scripts.


getLabelAnnotationsList is returing empty map

In which file did you encounter the issue?
java-docs-samples/vision/cloud-client/src/main/java/com/example/vision/QuickstartSample.java
Did you change the file? If so, how?
no. running on Tomcat 7 in java 1.8
Describe the issue
EntityAnnotation annotation : res.getLabelAnnotationsList()
there are no items in that. It was working fine some time back, now stopped working. I'm using:
com.google.cloud
google-cloud-vision
0.17.1-beta

For now gRPC does not support Tomcat well. We should update the doc of supported env https://github.com/GoogleCloudPlatform/google-cloud-java/issues/2039

I'll see if what I can repro on Tomcat, for my tests, I'm always seeing labels returned from label annotation queries.

@gguuss did you get a chance to repro?

I was actually unable to repro. The following code successfully performs a request from a JSP on tomcat.  All you need to do is put a jar containing all the client library dependencies in to the tomcat /lib folder from where you're serving (e.g. mvn compile assembly:single for the cloud-client sample and copy target/vision-google-cloud-samples-1.0.0-jar-with-dependencies.jar into /lib/ for your Tomcat directory.  Of note, I'm using Tomcat 8.5.14 installed from homebrew on OSX.
<%@ page import="com.google.cloud.vision.spi.v1.ImageAnnotatorClient, com.google.cloud.vision.spi.v1.ImageAnnotatorSettings, com.google.cloud.vision.v1.AnnotateImageRequest, com.google.cloud.vision.v1.AnnotateImageResponse, com.google.cloud.vision.v1.BatchAnnotateImagesResponse, com.google.cloud.vision.v1.Block, com.google.cloud.vision.v1.ColorInfo, com.google.cloud.vision.v1.CropHint, com.google.cloud.vision.v1.CropHintsAnnotation, com.google.cloud.vision.v1.DominantColorsAnnotation, com.google.cloud.vision.v1.EntityAnnotation, com.google.cloud.vision.v1.FaceAnnotation, com.google.cloud.vision.v1.Feature, com.google.cloud.vision.v1.Feature.Type, com.google.cloud.vision.v1.Image, com.google.cloud.vision.v1.ImageSource, com.google.cloud.vision.v1.LocationInfo, com.google.cloud.vision.v1.Page, com.google.cloud.vision.v1.Paragraph, com.google.cloud.vision.v1.SafeSearchAnnotation, com.google.cloud.vision.v1.Symbol, com.google.cloud.vision.v1.TextAnnotation, com.google.cloud.vision.v1.WebDetection, com.google.cloud.vision.v1.WebDetection.WebEntity, com.google.cloud.vision.v1.WebDetection.WebImage, com.google.cloud.vision.v1.WebDetection.WebPage, com.google.cloud.vision.v1.Word, com.google.protobuf.ByteString, org.threeten.bp.Duration, java.io.FileInputStream, java.io.IOException, java.io.PrintStream, java.util.ArrayList, java.util.List" %>
<% String gcsPath = "gs://cloud-samples-tests/vision/face_no_surprise.jpg"; %>

    <title>
      I know JSP: <%=request.getServletContext().getServerInfo()%>
    </title>

Hello Vision!

    This is where the action happen.

Performing face detection on <%=gcsPath%>
<%
    List<AnnotateImageRequest> reqs = new ArrayList<>();

    ImageAnnotatorSettings.Builder imageAnnotatorSettingsBuilder =
        ImageAnnotatorSettings.defaultBuilder();
    imageAnnotatorSettingsBuilder
        .batchAnnotateImagesSettings()
        .getRetrySettingsBuilder()
        .setTotalTimeout(Duration.ofSeconds(30));
    ImageAnnotatorSettings settings = imageAnnotatorSettingsBuilder.build();

    ImageSource imgSource = ImageSource.newBuilder().setGcsImageUri(gcsPath).build();
    Image img = Image.newBuilder().setSource(imgSource).build();
    Feature feat = Feature.newBuilder().setType(Type.FACE_DETECTION).build();

    AnnotateImageRequest req =
        AnnotateImageRequest.newBuilder().addFeatures(feat).setImage(img).build();
    reqs.add(req);

    ImageAnnotatorClient client = ImageAnnotatorClient.create(settings);
    BatchAnnotateImagesResponse resp = client.batchAnnotateImages(reqs);
    List<AnnotateImageResponse> responses = resp.getResponsesList();

    for (AnnotateImageResponse res : responses) {
      if (res.hasError()) {
        %>Error!!!! <%=res.getError().getMessage()%><%
        break;
      }

      // For full list of available annotations, see http://g.co/cloud/vision/docs
      for (FaceAnnotation annotation : res.getFaceAnnotationsList()) {
        %>
Anger: <%=annotation.getAngerLikelihood()%><%
        %>
Joy: <%=annotation.getSurpriseLikelihood()%><%
        %>
Poly: <%=annotation.getBoundingPoly()%><%
      }
    }
%>

Closing out since we are unable to repro.


[BUG]: ColossalAI/examples/images/diffusion/—The results of the sampling were poor

🐛 Describe the bug
The results of the sampling were poor
plms sampling：

dpm sampling：

Train 3000 epochs using the TEYVAT dataset
Environment
dpm or plms sampling
python txt2img.py --prompt "Teyvat, Name:Keqing, Element:Electro, Weapon:Sword, Region:Liyue, Model type:Medium Female, Description:an anime character wearing a purple dress and cat ears."
--dpm
--outdir ./output/fashion
--ckpt /home/project/ColossalAI/examples/images/diffusion/logfiles/2023-02-13T01-50-34_train_colossalai_teyvat/diff_tb/version_0/checkpoints/epoch=2917-step=11672.ckpt
--config /home/project/ColossalAI/examples/images/diffusion/logfiles/2023-02-13T01-50-34_train_colossalai_teyvat/configs/2023-02-13T01-50-34-project.yaml
--n_samples 2

I don't know what caused it. I tried many different trainings, PLMS sampling was always bad, DPM was a little better, but still there was noise. Looking forward to your reply! Thank you!

I will try this out myself and get back to you within the next two days.

How many steps have you trained?

@FrankLeeeee
I train 3000 epochs using the TEYVAT dataset（https://huggingface.co/datasets/Fazzie/Teyvat）, I just modified the configuration file'max_epochs（train_colossalai_teyvat.yaml).

Based on the last generated ckpt file(epoch=2917-step=11672.ckpt )，steps should be 11672.
At this step I saw that the loss was basically not optimized, so I stopped training at epoch 2917.

Is there any update on this? @JThh

Hi @LhaoH @Thomas2419 Diffusion convergence is a challenging task, and Colossal-AI provides a more effective development tool. This issue does not seem to be a bug caused by Colossal-AI. For personalized development issues, this goes beyond the scope of open source community support.
Feel free to contact me via email<EMAIL_ADDRESS>to discuss formal collaboration, and you will receive professional high-priority support to help you get your product development done quickly and at low cost. Thanks.
This issue was closed due to inactivity. Thanks.

@LhaoH Your procedure is basically for fine-tuning the model which won't make meaningful improvement directly on pretrained checkpoints, due to the limited dataset size. Have you run the training procedure using LAION dataset?

I did not try it on the LAION dataset, but I ran 2 epochs on the custom dataset (14w), totaling 5054 steps, and the generation effect was relatively poor.
prompt:"with slightly puffed shoulder and a flattering u neckline this cozy long sleeved top is easy to wear and effortlessly feminine"

prompt:"sharp vintage frame define stylish sunglass with indelible appeal"

Because I am unable to solve this problem,So I have given up using colosalai. Currently, but I have successfully run the official fine-tuning code on my own dataset. Thank you for keeping this issue in mind.Thank you very much.


added boostrap for svelte repo

Thank you! I'll merge this when v3 compatibility lands - see https://github.com/bestguy/sveltestrap/pull/15 and https://github.com/bestguy/sveltestrap/pull/16


[tool] connected IOS devices not visible on old version of Xcode, "Flutter requires Xcode 13 or higher"

Sorry for the trouble.
It work's fine in flutter version 2.10.5 stable, but when I upgrade my flutter environment to 3 this problem occur, even though the emulator is already open.
But when I will downgrade the flutter version to  2.10.5 it will work again.
Btw I'm only using Xcode 12.4 because I have an old mac and I can't upgrade to Xcode 13 or higher.

Hi @lel101, Thanks for filing the issue. Can you please share the output of flutter doctor -v on both stable 2.10 .5 (where it works ) and stable 3.0 (where it doesn't)?

@maheshmnj Thank you for your response
stable 3.0
-- flutter doctor
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 3.0.0, on Mac OS X 10.15.7 19H1922 darwin-x64, locale en-PH)
[✓] Android toolchain - develop for Android devices (Android SDK version 32.1.0-rc1)
[!] Xcode - develop for iOS and macOS (Xcode 12.4)
    ✗ Flutter requires Xcode 13 or higher.
      Download the latest version or update via the Mac App Store.
[✓] Chrome - develop for the web
[✓] Android Studio (version 2021.1)
[✓] VS Code (version 1.67.2)
[✓] VS Code (version 1.66.2)
[✓] VS Code (version 1.66.2)
[✓] Connected device (2 available)
[✓] HTTP Host Availability

! Doctor found issues in 1 category.

-- flutter devices
2 connected devices:

macOS (desktop) • macos  • darwin-x64     • Mac OS X 10.15.7 19H1922 darwin-x64
Chrome (web)    • chrome • web-javascript • Google Chrome 101.0.4951.64

stable 2.10.5
-- flutter doctor
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 2.10.5, on Mac OS X 10.15.7 19H1922 darwin-x64, locale en-PH)
[✓] Android toolchain - develop for Android devices (Android SDK version 32.1.0-rc1)
[!] Xcode - develop for iOS and macOS (Xcode 12.4)
    ! Flutter recommends a minimum Xcode version of 13.
      Download the latest version or update via the Mac App Store.
[✓] Chrome - develop for the web
[✓] Android Studio (version 2021.1)
[✓] VS Code (version 1.67.2)
[✓] VS Code (version 1.66.2)
[✓] VS Code (version 1.66.2)
[✓] Connected device (2 available)
[✓] HTTP Host Availability

! Doctor found issues in 1 category.

-- flutter devices
2 connected devices:

Custom_Simulator (mobile) • 23F99BB7-0559-4832-8072-091C76495A88 • ios            • com.apple.CoreSimulator.SimRuntime.iOS-14-4
(simulator)
Chrome (web)              • chrome                               • web-javascript • Google Chrome 101.0.4951.64

Thanks for the info @lel101,

✗ Flutter requires Xcode 13 or higher.

I believe you would need to  upgrade xcode to 13, However, you can try this solution https://github.com/flutter/flutter/issues/41006#issuecomment-533748314 and see if it helps fix the issue  ?

@maheshmnj
Thanks for the response,

I believe you would need to upgrade xcode to 13
I'm using old mac that's why I cannot upgrade my xcode :(

However, you can try this solution https://github.com/flutter/flutter/issues/41006#issuecomment-533748314 and see if it helps fix the issue ?
I will check this one thanks!

Thanks for the info @lel101,

✗ Flutter requires Xcode 13 or higher.

I believe you would need to upgrade xcode to 13, However, you can try this solution #41006 (comment) and see if it helps fix the issue ?

I got same error too.. i have tried this.. but still not work in flutter 3. please confirm @lel101 have u tried it?

@AbiMulLyA
Sorry but still not working. Do you have same setup with me ?
What version of xcode you are using ?

@lel101  i'm using xcode 12.4 and currently on flutter 3.0.1

@AbiMulLyA
Can you try upgrade your xcode to 13 and test it ?
Sorry I wish I could try it here but my laptop is old and I can't upgrade it.

@lel101 sorry i'm using old mac too. I can't update to xcode 13 😆

@maheshmnj
Sorry for the trouble,
Do you have any work around for laptop that uses xcode below 13 ?

@maheshmnj  is there another solution for this?

Thanks for the info @lel101, I am not sure of any other solution to why the ios devices are not visible, The xcode version seems to be the cause of the issue. Labeling this issue for further insights from the team.

cc: @christopherfujino

I was just thinking about this in https://github.com/flutter/flutter/issues/103844#issuecomment-1133150030
We could change the Xcode version logic from being so aggressive--only check if it's installed, then try to build or get devices or whatever, and only check and report the version if that fails.  The last Xcode bump, for example, was because we need bitcode-enabled apps (usually add-to-app) to be on the same or newer version of clang than the one the engine is on.  But for most of our users that's not relevant (bitcode is off in the template).  Sometimes it just won't build on an old toolchain, at which point we can report that the version is too old.

App Store does not accept archives built with Xcode < 13 anymore. I have a Mac Mini late 2012 and it doesn't support macOS Monterey. I got Monterey installed using the OpenCore Legacy Patcher and then I was able to install Xcode 13 without issues. You can check supported models here.

That's one of the reasons we're enforcing it so strictly on flutter build/flutter run.
Ideally everyone should be on the latest and greatest version of Xcode with the newest SDKs.  In practice though it can be really difficult to, say, upgrade your entire CI suite hardware, when it only takes one new machine (or deployment service) to actually archive and publish.  We could allow looser build/run requirements, but be stricter for flutter build ipa when developers are presumably ready to ship their app.

@lel101  could u run your project on iPhone real device with current flutter n xcode version?

On the latest versions, Flutter requires Xcode 14 or higher as included in the 3.13.0 release note.
Running flutter doctor -v with XCcde 13, gives:
[✓] Flutter (Channel stable, 3.13.2, on macOS 11.7.9 20G1426 darwin-x64, locale pl-PL)
    • Flutter version 3.13.2 on channel stable at /Users/<user_name>/development/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision ff5b5b5fa6 (13 days ago), 2023-08-24 08:12:28 -0500
    • Engine revision b20183e040
    • Dart version 3.1.0
    • DevTools version 2.25.0

[✗] Android toolchain - develop for Android devices
    ✗ Unable to locate Android SDK.
      Install Android Studio from: https://developer.android.com/studio/index.html
      On first launch it will assist you in installing the Android SDK components.
      (or visit https://flutter.dev/docs/get-started/install/macos#android-setup for detailed instructions).
      If the Android SDK has been installed to a custom location, please use
      `flutter config --android-sdk` to update to that location.

[!] Xcode - develop for iOS and macOS (Xcode 13.0)
    • Xcode at /Users/<user_name>/Downloads/Xcode 2.app/Contents/Developer
    • Build 13A233
    ✗ Flutter requires Xcode 14 or higher.
      Download the latest version or update via the Mac App Store.
    ✗ CocoaPods not installed.
        CocoaPods is used to retrieve the iOS and macOS platform side's plugin code that responds to your plugin usage on the Dart side.
        Without CocoaPods, plugins will not work on iOS or macOS.
        For more info, see https://flutter.dev/platform-plugins
      To install see https://guides.cocoapods.org/using/getting-started.html#installation for instructions.

[✗] Chrome - develop for the web (Cannot find Chrome executable at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome)
    ! Cannot find Chrome. Try setting CHROME_EXECUTABLE to a Chrome executable.

[!] Android Studio (not installed)
    • Android Studio not found; download from https://developer.android.com/studio/index.html
      (or visit https://flutter.dev/docs/get-started/install/macos#android-setup for detailed instructions).

[✓] VS Code (version 1.81.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.72.0

[✓] Connected device (1 available)
    • macOS (desktop) • macos • darwin-x64 • macOS 11.7.9 20G1426 darwin-x64

[✓] Network resources
    • All expected network resources are available.

! Doctor found issues in 4 categories.

/cc @christopherfujino

@dam-ease why are you tagging me?

I saw a similar issue recently on the new Flutter version and error message, so I tagged you because I saw you add a priority label and thought you might want to follow up on this issue by keeping up to date with the latest information. Apologies if my assumption is incorrect.

I saw a similar issue recently on the new Flutter version and error message, so I tagged you because I saw you add a priority label and thought you might want to follow up on this issue by keeping up to date with the latest information. Apologies if my assumption is incorrect.

this approach does not scale well to the number of open issues that are labelled tool: https://github.com/flutter/flutter/issues?q=is%3Aopen+is%3Aissue+label%3Atool. Per our Issue Hygiene section of the wiki, I would recommend adding a thumbs up reaction to the top level comment of the issue in order to raise visibility of it.


Pagination for table

First of all, thanks for this awesome work!
It would be great if the component also has pagination for results.
Is their any method to support pagination?

I second this


Fix executions with assertions on circular objects

Fixes #193.

@vtintillier wrote at issue 193:

Created #195 , but no CLA prompt yet.

Looks like something stopped working. No problem, we can do it manully here.

Developer's Certificate of Origin 1.1 https://developercertificate.org/
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

Please read the above and reply stating whether the above is true in your case. In a nut shell, we are asking whether you wrote the patch yourself and that you are able and willing to release it under the MIT open-source license.

Please read the above and reply stating whether the above is true in your case. In a nut shell, we are asking whether you wrote the patch yourself and that you are able and willing to release it under the MIT open-source license.

The above is true, I wrote the patch myself 👍