jest.setSystemTime is not working, but mockdate is

Hi there, I'm trying to use the fake timers from jest, but I can't get them to work. I'm using jest-light-runner with esmock, and my dates are handled by dayjs.
Non-working snippet:
const now = dayjs();

beforeAll(async () => {
  await db.raw(`ATTACH DATABASE ":memory:" AS ${Customer.schema}`);
  await db.migrate.latest();
  jest.useFakeTimers();
  jest.setSystemTime(now);
});

Resulting in:
          OptIn {
    -       "effectiveAt": "2023-06-30T08:47:48.437Z",
    +       "effectiveAt": "2023-06-30T08:47:48.463Z",
            "name": "text",
            "value": true,
          },

Working snippet:
const now = dayjs();

beforeAll(async () => {
  await db.raw(`ATTACH DATABASE ":memory:" AS ${Customer.schema}`);
  await db.migrate.latest();
  MockDate.set(now);
});

Am I doing something wrong? I also tried legacy timers in jest to no avail.

What version are you on? Fake timers support was only published a few weeks ago: https://github.com/nicolo-ribaudo/jest-light-runner/releases/tag/v0.5.0

I am on 0.5.0, here are my dev deps:
"devDependencies": {
    "@babel/core": "^7.22.5",
    "@babel/eslint-parser": "^7.22.5",
    "@babel/plugin-syntax-import-assertions": "^7.22.5",
    "babel-jest": "^29.5.0",
    "better-sqlite3": "^8.4.0",
    "c8": "^8.0.0",
    "clinic": "^12.1.0",
    "esmock": "^2.3.1",
    "healthier": "^6.3.0",
    "jest": "^29.5.0",
    "jest-junit": "^16.0.0",
    "jest-light-runner": "^0.5.0",
    "mockdate": "^3.0.5",
    "nodemon": "^2.0.22",
    "prettier": "^2.8.8",
    "supertest": "^6.3.3"
  },

setSystemTime takes a Date object, not a Dayjs object. The correct usage is this.
  jest.useFakeTimers().setSystemTime(dayjs().toDate());

You would have caught this error if you would have used TypeScript:

Me and 2xic (feature author) have used the features released in 0.5.0 in our own larger codebase for 6 months now. Below are some code snippets and functions that have served us well.
jest.setup.after.env.js
const dayjs = require('dayjs');

expect.addSnapshotSerializer({
  test: (arg) => dayjs.isDayjs(arg),
  print: (val) => {
    const content = val.isSame(val.startOf('day')) ? val.format('YYYY-MM-DD') : val.toISOString();

    return '"' + content + '"';
  },
});

setMockTime.ts
import dayjs, { Dayjs } from 'dayjs';

/**
 * Be sure to call jest.useRealTimers() in afterEach when using this, or the pipeline will time out after 1 hour.
 */
export function setMockTime(dateTime: string | Dayjs): Dayjs {
  const dayjsObject = dayjs(dateTime, { utc: true }).tz('Europe/Oslo'); // your timezone

  // standard validation of dayjs objects
  try {
    dayjsObject.toISOString();
  } catch (e) {
    throw new Error(`Invalid date string: "${dateTime}"`);
  }

  jest.useFakeTimers().setSystemTime(dayjsObject.toDate());

  return dayjsObject;
}

setMockTime.test.ts
import dayjs from 'dayjs';
import { setMockTime } from './setMockTime';

describe('useFakeTimers', () => {
  afterEach(() => {
    jest.useRealTimers();
  });

  it('should set the time to a constant value', async () => {
    setMockTime('2023-01-01 12:00');

    expect(dayjs()).toMatchInlineSnapshot(`"2023-01-01T12:00:00.000Z"`);
  });

  it('should throw error if we try to set the time to an invalid time string', () => {
    expect(() => setMockTime('2023-01--01 12:00:00')).toThrowErrorMatchingInlineSnapshot(
      `"Invalid date string: "2023-01--01 12:00:00""`,
    );
  });
});

This can be closed if the fix works for @alucryd

Thank you! I'll reopen if the fix does not work.

@nicolo-ribaudo Using a Date object does indeed work, thanks! However I'm now running into another issue, the timer logic works fine for a single test file, however if I try to put the following block in multiple files, subsequent calls to jest.useFakeTimers() will completely freeze the tests, only the first file will run, then the terminal hangs indefinitely.
beforeAll(async () => {
  await db.migrate.latest();
  jest.useFakeTimers().setSystemTime(now.toDate());
});

afterAll(async () => {
  await db.destroy();
  jest.useRealTimers();
});

Yes, i have also seen the bug you are describing. My current workound is to be sure to call jest.useRealTimers() in afterEach when i am using fake timers. I hinted at this in the docstring of my setMockTime function above. We usually don't mock the time in every test, but only when the test relies on the time to pass. Most of your tests should work without mocking the time.
@alucryd Could you create a new issue with a minimal reproducible example and instructions of how to execute this bug? It is better to fix this in a new issue than to reuse this closed issue.


Support Python 3.11

Now that Pastas supports it (https://github.com/pastas/pastas/pull/588), Pastastore should be fine as well

:(
ERROR: Could not find a version that satisfies the requirement arcticdb; extra == "arcticdb" (from pastastore[arcticdb,lint,optional]) (from versions: none)
ERROR: No matching distribution found for arcticdb; extra == "arcticdb"

Since ArtcticDB v1.1.0 that supports py311 this works as well.


AbstractLiteral toString is comparing a CoreDatatype with an IRI

Current Behavior
at the line 195 we have datatype.equals(CoreDatatype.XSD.STRING, but datatype is of type IRI and CoreDatatype.XSD.STRING of type CoreDatatype, making it false all the time.
https://github.com/eclipse/rdf4j/blob/78771b7161f6b5a84b237db597b53a8842935860/core/model-api/src/main/java/org/eclipse/rdf4j/model/base/AbstractLiteral.java#L195
Making the toString value of an IRI leading to that:
"abdc"^^<http://www.w3.org/2001/XMLSchema#string>

Expected Behavior
printing the literal without the XSD.STRING datatype like with all the other version of the toString
"abdc"

Steps To Reproduce
I'm using a custom ValueFactory using the Abstract one, but this one is also doing it:
System.out.println(
  new AbstractValueFactory() {}
     .createLiteral(("abcd"))
);

Version
4.2.3
Are you interested in contributing a solution yourself?
Yes
Anything else?
No response

Thanks @ate47 , good find!


[filters] Add RBAC condition fuzzer (CEL expressions)

This adds a fuzzer for CEL expression matching. These conditions are used in the RBAC filter for complementing the existing principal/permission model.
About a quarter of the execution time is coming from google::api::expr::runtime::RegisterBuiltinFunctions. The test runs locally at about 250 exec/sec.
See: #7716
Risk Level: Low
Testing: Converted unit tests into corpus entries
Re-open of #8752 -- all history the same, merged to up to date master, and pushed all changes.
/review @htuch

Thanks! @tswadell FYI.


Inspector: Add missing support for inspectableCustomProperties

See https://forum.babylonjs.com/t/light-scan-postprocess/38816/5

Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s).
To prevent this PR from going to the changelog marked it with the "skip changelog" label.

Snapshot stored with reference name:
refs/pull/13599/merge
Test environment:
https://babylonsnapshots.z22.web.core.windows.net/refs/pull/13599/merge/index.html
To test a playground add it to the URL, for example:
https://babylonsnapshots.z22.web.core.windows.net/refs/pull/13599/merge/index.html#WGZLGJ#4600
Links to test babylon tools with this snapshot:
https://playground.babylonjs.com/?snapshot=refs/pull/13599/merge
https://sandbox.babylonjs.com/?snapshot=refs/pull/13599/merge
https://gui.babylonjs.com/?snapshot=refs/pull/13599/merge
https://nme.babylonjs.com/?snapshot=refs/pull/13599/merge
To test the snapshot in the playground with a playground ID add it after the snapshot query string:
https://playground.babylonjs.com/?snapshot=refs/pull/13599/merge#BCU1XR#0


Multiple poly in trace

Multiple poly in trace
Description
Description of the pull request changes and motivation.
Type of change
Please delete options that are not relevant.

[ ] New feature
[ ] Bug fix
[ ] Optimization

Checklist

[ ] Linked to Github Issue
[ ] Unit tests added
[ ] This change requires new documentation.

[ ] Documentation has been added/updated.

[ ] This change is an Optimization

[ ] Benchmarks added/run

Codecov Report

Merging #206 (1d3e023) into main (3fc6fdb) will increase coverage by 0.04%.
The diff coverage is 93.21%.

@@            Coverage Diff             @@
##             main     #206      +/-   ##
==========================================
+ Coverage   92.17%   92.21%   +0.04%
==========================================
  Files          66       66
  Lines        7208     7310     +102
==========================================
+ Hits         6644     6741      +97
- Misses        564      569       +5

Impacted Files
Coverage Δ

proving_system/stark/src/air/mod.rs
66.66% <ø> (ø)

proving_system/stark/src/lib.rs
58.85% <56.25%> (+0.47%)
:arrow_up:

proving_system/stark/src/air/trace.rs
84.90% <84.61%> (-15.10%)
:arrow_down:

...oving_system/stark/src/air/constraints/boundary.rs
90.62% <100.00%> (+2.01%)
:arrow_up:

...ving_system/stark/src/air/constraints/evaluator.rs
100.00% <100.00%> (ø)

proving_system/stark/src/air/frame.rs
81.96% <100.00%> (-0.85%)
:arrow_down:

proving_system/stark/src/prover.rs
100.00% <100.00%> (ø)

proving_system/stark/src/verifier.rs
98.10% <100.00%> (+0.20%)
:arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


One time login without save it session to browser

When login using Google Client, a session / cookie data of current user account is saved to the browser, how to prevent it (implement one time login without save it session / cookie to browser) ?
And Is there a function to logout from Google using Google Client API (clear logged user's session / cookie) ?

Hi @murzid,
Can you share the code you're using? The client itself only handles providing auth tokens, not persistence. Are you following a tutorial or example which sets cookies?

// Google Client
$client = new Google_Client();
$client->setApplicationName('MY APPS');
$client->setClientId('xxxxxxx.apps.googleusercontent.com');
$client->setClientSecret('xxxxxxxxxxxxxxxxxxxxxxx');
$client->setRedirectUri('http://domain.com/login');
$client->setAccessType('offline');
$client->setApprovalPrompt('force');
$client->setScopes([
	"https://www.googleapis.com/auth/userinfo.email",
	"https://www.googleapis.com/auth/userinfo.profile",
	"https://www.googleapis.com/auth/plus.me"
]);

// Google Auth URL
$url = $client->createAuthUrl();

// Login Link
echo 'Click here to login';

After successfully entering the Google Account, we can automatically access all Google services (such as Gmail, photos, drives, etc.) on other browser tabs.
Even though what we want to do is just log in to the website http://domain.com, not to access Google services on other browser tabs.
After closing the browser and reopening it, and then logging into http://domain.com we are not asked to enter a Google Account again because the data has been stored in the browser (session / cookie created by Google).
Can this problem be prevented?

Since you must be logged into a Google account to obtain an auth token for your application, the login status on Google is controlled by Google, not by your application. Unfortunately, there's not much you can do to change this as the application developer. Your users could mitigate this by using a private browser session, for instance (which would clear stored cookies once closed), though this would also impact the login state of your application.
I believe that should resolve the issue. If I've misunderstood or not fully answered your question, please reopen and we'll continue working on a solution. :)


Optional dates can’t use Field()

Checks

[x] I added a descriptive title to this issue
[x] I have searched (google, github) for similar issues and couldn't find anything
[x] I have read and followed the docs and still think this is a bug

Bug
Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":
        pydantic version: 1.8.2
       pydantic compiled: False
            install path: /home/www/.cache/pypoetry/virtualenvs/redacted-ydm62z-a-py3.10/lib/python3.10/site-packages/pydantic
          python version: 3.10.0 (default, Nov 18 2021, 00:02:14) [GCC 10.2.1 20210110]
                platform: Linux-5.10.47-linuxkit-aarch64-with-glibc2.31
optional deps. installed: ['dotenv', 'email-validator', 'typing-extensions']

from typing import Optional
from datetime import date
from pydantic import BaseModel, Field

class SuccessModelOne(BaseModel):
	floatTest: Optional[float]
	strTest: Optional[str]
	date: Optional[date]

class SuccessModelTwo(BaseModel):
	floatTest: Optional[float] = Field(description='This is a float test')
	strTest: Optional[str] = Field(description='This is a str test')

class FailureModel(BaseModel):
	date: Optional[date] = Field(description='This is a date test')

The success models work as expected. The failure model raises the following exception:
TypeError: typing.Optional requires a single type. Got FieldInfo(default=PydanticUndefined, description='This is a date test', extra={}).

It seems like other field types are fine, but specifically optional datetime.date properties can’t have Field()s associated with them.
I checked the documentation and didn’t see anything relevant.

Hi there, I would like to work on this issue.

Is there a workaround for this issue?

I isn't
Sorry :(

Hi
It has nothing to do with pydantic. You can reproduce the error with this
from typing import Optional
from datetime import date

date: Optional[date] = date.today()

TypeError: typing.Optional requires a single type. Got datetime.date(2022, 1, 12).

If you want your field to be named "date", you can either use an alias for the import (which I recommend)
...
from datetime import date as dtdate

class M(BaseModel):
	date: Optional[dtdate] = Field(description='This is a date test')

or rely on the alias parameter in pydantic
from datetime import date

class M(BaseModel):
	thedate: Optional[date] = Field(alias='date', description='This is a date test')

Hope it helps!

This code is functional without workarounds (if you don't count the import...):
from __future__ import annotations

from datetime import date
from typing import Optional

from pydantic import BaseModel, Field

class M(BaseModel):
    date: Optional[date] = Field(description="This is a date test")

See also PEP 563.

I just stumbled upon this error on some code generated by the OpenAPI Python client generator.
The generated code looks just like the sample given by @alexpovel above (with __future__.annotations import) but a TypeError is raised anyway.
If anyone is in this case and wants to avoid renaming the property in the OpenAPI definition, using the option --name-mappings of the generator seems like the best option.

For completeness, I executed the code sample in the Python docker official images since 3.7 (appearance of __future__.annotations with PEP 563). Here are the different error raised:
# Python 3.7 and 3.8
TypeError: Optional[t] requires a single type. Got FieldInfo(annotation=NoneType, required=True, description='This is a date test').

# Python 3.9 and 3.10
TypeError: typing.Optional requires a single type. Got FieldInfo(annotation=NoneType, required=True, description='This is a date test').

# Python 3.11 and 3.12
RecursionError: maximum recursion depth exceeded

The pydantic version is 2.7.1 except for Python 3.7, in which case it is 2.5.3.


Show info message box when terminal is paused

Would be nice to have some type of info message box that informs that the current terminal tab is paused.
Eg. KDE Konsole has this info box:

I don't think the suspension of the foreground process is a terminal thing, but a shell thing (Bash, specifically). My understanding is that Terminal doesn't know what CTRL+S means, it just gives that to the shell it's attached to, unless it is specifically bound to some action in Terminal.  So, unless bash can tell the terminal emulator that the foreground process is suspended, I don't know how this would be possible.

This is about suspending output (Ctrl+S, XOFF, termios::c_cc[VSTOP]), not about suspending a process (Ctrl+Z, SIGTSTP, termios::c_cc[VSUSP]).
The console host suspends output if you press the Pause key (see InputBuffer::_HandleConsoleSuspensionEvents), but Ctrl+S does not currently have the same effect (https://github.com/microsoft/terminal/issues/809). The terminal could be made to show an info message about this, but it doesn't seem very necessary because pressing almost any key resumes output anyway.
If you see Ctrl+S suspend output from Windows Terminal, then that's handled by an application or by the tty driver of a remote computer, and AFAIK there is no standard way for an application to tell the terminal that it handled the Ctrl+S keypress in this way; neither in the Win32 console API nor over a serial connection.
I don't know how KDE Konsole is implemented but I suspect that it monitors TIOCPKT_STOP and TIOCPKT_START events from a pseudoterminal device. Does that work also if you run ssh from Konsole?

I'm not using it over ssh that is an extreme case, but I'm using it heavily when I'm compiling something or invoking clang-tidy or clazy and want to pause it because I need to free CPU.
I have tried it now and it works the same way in Konsole and in the wterm, so eg. if some script forks 10 processes eg. clang-tidy (invoked by run-clang-tidy.py) or clang++14 during compilation and you press CTRL+s in Konsole or Pause in wterm, then all that 10 processes are still running and finish, but no new processes will be spawned.
After unpause the execution continues and a new clang-tidy or clang++-14 processes will be spawned.
Another example that I have tried, eg. if I pause during cmake configure then everything that is currently executed finishes and nothing new is executed (many things are happening during the cmake configure step), when I unpause the rest of the output is replayed to the console and configuration continues, practically the same what I described in the paragraph above.
It looks like it only pauses a direct child and because of it the Konsole's message states that the Output was suspended.
In wterm and also in Konsole the rest of the output during this suspension is recorded and after unpause is printed to the console or replayed.
@KalleOlaviNiemitalo You are right on windows you can unpaused by any key, but it doesn't matter how it is paused or unpaused by which keys, problem is that when I pause then I don't have any visual feedback that the output is or was paused.

This is a cool idea, for sure, but we'd need some way for the Terminal to be able to detect that the output was suspended. If someone can figure out how to do that, then I'm all for it. I'm not sure that WSL (or ssh.exe) expose that information, however. I'm pretty sure this would end up requiring a much larger change to how signaling is done on Windows 😅

https://www.rfc-editor.org/rfc/rfc4254#section-6.8 SSH_MSG_CHANNEL_REQUEST "xon-xoff" is a way for an SSH server to ask the SSH client to handle Ctrl+S and Ctrl+Q from the terminal. I imagine the SSH client could then tell the terminal (perhaps via the console API, if you still dared to extend that) that it has done so. https://github.com/PowerShell/openssh-portable/ does not contain the "xon-xoff" string though, not even at the SSH server side.
But if this issue is only about the Pause key for which the console host already has a flag, then showing an info bar for that should be easier than doing the same for flow control with a remote computer.

The wiki page about Software flow control and XON/XOFF codes.
For me is enough to detect the Pause and CTRL+S keys for pause and CTRL+Q for resume, or it can stay like it is now, so every key press resumes.
I have looked how the Konsole is doing it and it simply detects key presses in the Vt102Emulation::sendKeyEvent() handler. Main connect() is here, signals emited here and events handled in the Session::updateFlowControlState().  The XON/XOFF are sent from the event handler using the flowControlEnabled() it internally calls the Pty::flowControlEnabled().
The same can be done in wterm, I don't know if wterm uses XON/XOFF to pause the terminal but it doesn't matter. wterm already has code that is catching the PAUSE key press, much can be found by searching the pause word on GitHub here.
The code handling the pause for wterm is in the InputBuffer::_HandleConsoleSuspensionEvents().

I see that more of you are writing that will be needed to detect that the terminal is suspended, this is already done in the wterm, wterm is already pausing the terminal, and nothing new code is required, he is already pausing and unpausing the terminal output!
Only one thing that is needed to write, are methods to display and hide the info box during the pause and unpause. 😎

<off topic showerthought> @Tyriar this might make an interesting Quick Tip for the vscode terminal, if there's a good way of wiring that up. IK when I was first getting the commandline ropes I'd definitely instinctively Ctrl+S in vim and be super confused when it stopped working.


[Bug] FlatMapperIterator should be closed after usage

Bug Type (问题类型)
others (please edit later)
Before submit

[X] 我已经确认现有的 Issues 与 FAQ 中没有相同 / 重复问题 (I have confirmed and searched that there are no similar problems in the historical issue and documents)

Environment (环境信息)

SonarQube block issue [java:S2095]

Connections, streams, files, and other classes that implement the Closeable interface or its super-interface, AutoCloseable, needs to be closed after use. Further, that close call must be made in a finally block otherwise an exception could keep the call from being made. Preferably, when class implements AutoCloseable, resource should be created using "try-with-resources" pattern and will be closed automatically.
Failure to properly close resources will result in a resource leak which could bring first the application and then perhaps the box the application is on to their knees.
Expected & Actual behavior (期望与实际表现)
        // Multiple steps, construct first step to iterator
        Iterator<Edge> edges = this.edgesOfVertexWithCount(source, firstStep);
        // Wrap steps to Iterator except last step
        for (int i = 1; i < stepNum - 1; i++) {
            EdgeStep currentStep = steps.get(i);
            edges = new FlatMapperIterator<>(edges, (edge) -> {
                Id target = ((HugeEdge) edge).id().otherVertexId();
                return this.edgesOfVertexWithCount(target, currentStep);
            });
        }

close FlatMapperIterator after usage
Vertex/Edge example (问题点 / 边数据举例)
No response
Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)
No response

There are indeed some iterators in the current code that have not been finally closed.
U can confirm their lifecycle, if they are indeed legacy and may cause potential impact, you can help submit PR fixes (there should be some places in the class)

There are indeed some iterators in the current code that have not been finally closed.
U can confirm their lifecycle, if they are indeed legacy and may cause potential impact, you can help submit PR fixes (there should be some places in the class)

I have scanned the code, this is only case and u can review the above PR.


manual testing for Sprint 6

TESTING.md
TESTING.md
cleanup for:

Ubuntu (VDI)
Mac

Windows

[ ] making hints on missing plugins #9
[ ] add new startup notification saying the plugin is in beta/dev #44
[ ] install required dependencies for the project from LOCAL #30
[ ] implement REPL thingy #92
[ ] persistant setting for REPL (show dialog)
[ ] Import IDE settings from remote #117
[ ] Import project settings from remote #118
[ ] Scala SDK is installed
[ ] removing O1 marks other modules invalid

Linux

[ ] making hints on missing plugins #9
[ ] add new startup notification saying the plugin is in beta/dev #44
[ ] install required dependencies for the project from LOCAL #30
[ ] implement REPL thingy #92
[ ] persistant setting for REPL (show dialog)
[ ] Import IDE settings from remote #117
[ ] Import project settings from remote #118
[ ] Scala SDK is installed
[ ] removing O1 marks other modules invalid

Mac

[ ] making hints on missing plugins #9
[ ] add new startup notification saying the plugin is in beta/dev #44
[ ] install required dependencies for the project from LOCAL #30
[ ] implement REPL thingy #92
[ ] persistant setting for REPL (show dialog)
[ ] Import IDE settings from remote #117
[ ] Import project settings from remote #118
[ ] Scala SDK is installed
[ ] removing O1 marks other modules invalid

Notes for Windows:
# Initialization

Generated .zip with ./gradlew buildPlugin on master

# Windows 10

## Setup

Windows 10 VDI
Installed IntelliJ IDEA
Installed plugin from disk
Downloaded Zulu OpenJDK 11 x64
New Project -> Java
    JDK set
Indexing takes a long time

## making hints on missing plugins #9

Restarted IDE
!!! NO SCALA PLUGIN NOTIFICATION !!!
Installed Scala plugin from marketplace
Restarted IDE
No Scala plugin notification (as expected)
Scala plugin disabled
Restarted IDE
!!! NO SCALA PLUGIN NOTIFICATION !!!
Enabled Scala plugin
Restarted IDE
No Scala plugin notification (as expected)

## add new startup notification saying the plugin is in beta/dev #44

The notification of that says plugin is under development is shown

## install required dependencies for the project from LOCAL #30

Created a new project
!! TESTING.md is out of date. !!
    Have to turn project into A+ project first
! The image in TESTING.md is missing lib directory !

## implement REPL thingy #92

!! TESTING.md is out of date. !!
    Have to turn project into A+ project first
    Step 2.2. should be removed
Running Scala REPL in the first time takes a long time
! Image in Step 2.3.iii doesn't match the instructions of the previous step !
! Part 2.4. is redundant (cf. 2.1.iv) !

## turn project into "A+ O1" project" feature #86

! More detailed steps would be useful !
    Especially, how to get to the initial state (reset settings)
!! There should be better instructions on how to execute step 11. !!
! What I should observe on step 12. !

## Renaming/removing module files leads to incorrect states in module list #110

!! TESTING.md is out of date. !!
    Have to turn project into A+ project first

## Renaming/removing module files leads to incorrect states in module list #110

!!! After part 3, the modules don't compile !!!

## Resetting A+ Courses Settings

! Have to start with a new project because the previous test left the project in a bad state !

Notes for Ubuntu
when turned to A+ project (settings not imported) the project is reloaded even though that might be unnecessary
the dialog "Adjust Intellij IDEA Settings" should explain more explicitely what's gonna happen
  !!! cancelling the dialog should do nothing !!!
Modules panel should be invisible until project is turned to an A+ project
UI enhancement idea: Icon for topmost A+ menu
What should happen when project is turned to an a+ project but restart is opted out
    (It seems that the project is reloaded still)
    The menu item should be grayed out. (Also after restart.)
Turning project into A+ project: "Restart or not?" What should closing the dialog (cancelling) do?
Removed scala sdk, reinstalled module, have to do that twice before they are marked installed.
    IML files are modifies when modules are removed.


Cannot write record after set record to public read write

Update access to public read write:
curl 'http://api.tailorm.dev/record/save' -H 'Pragma: no-cache' -H 'Origin: http://portal.tailorm.dev' -H 'Accept-Encoding: gzip, deflate' -H 'X-Skygear-Access-Token: 3e0add12-f662-4ae3-9936-c5737024816b' -H 'Accept-Language: en-US,en;q=0.8,zh-CN;q=0.6,zh;q=0.4,zh-TW;q=0.2' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36' -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'Cache-Control: no-cache' -H 'Referer: http://portal.tailorm.dev/booking/3b766adb-aca3-4cfe-87ea-590dadcc2c5e' -H 'Connection: keep-alive' -H 'X-Skygear-API-Key: tailorm-key' --data-binary '{"action":"record:save","api_key":"tailorm-key","access_token":"3e0add12-f662-4ae3-9936-c5737024816b","database_id":"_public","records":[{"_id":"TailormBooking/3b766adb-aca3-4cfe-87ea-590dadcc2c5e","_access":[{"public":true,"level":"write"}],"notes":""}]}' --compressed

Update access response:
{"result":[{"_access":[{"level":"write","public":true}],"_created_at":"2016-05-12T10:17:03.999031Z","_created_by":"a03266e3-9b73-476a-b1f0-c784bbf43a63","_id":"TailormBooking/3b766adb-aca3-4cfe-87ea-590dadcc2c5e","_ownerID":"a03266e3-9b73-476a-b1f0-c784bbf43a63","_type":"record","_updated_at":"2016-05-26T04:41:25.224553Z","_updated_by":"f3c547cf-130d-41bc-bc31-64f29dcfc18a","billing_address":{"address":"","aptNumber":"","billingCountry":"HK","city":"","firstName":"","lastName":"","postalCode":"","state":""},"booking_number":"160399","booking_number_seq":194,"braintree_customer_id":"","braintree_payment_method_nonce":"d2924c9c-bca9-4307-abe6-1c96ab0b7601","braintree_payment_method_token":"fmqkjr","is_review_given":false,"notes":"","price":8000,"price_confirmed":false,"ready":false,"remarks":"","status":"booked","tailor_product":{"$id":"TailormTailorProduct/ba6bc7f4-5ec0-4355-95d9-9d0616f2df48","$type":"ref"},"user":{"$id":"TailormUser/a03266e3-9b73-476a-b1f0-c784bbf43a63","$type":"ref"}}]}

User who is not the owner update the record again:
 curl 'http://api.tailorm.dev/record/save' -H 'Pragma: no-cache' -H 'Origin: http://portal.tailorm.dev' -H 'Accept-Encoding: gzip, deflate' -H 'X-Skygear-Access-Token: 3e0add12-f662-4ae3-9936-c5737024816b' -H 'Accept-Language: en-US,en;q=0.8,zh-CN;q=0.6,zh;q=0.4,zh-TW;q=0.2' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36' -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'Cache-Control: no-cache' -H 'Referer: http://portal.tailorm.dev/booking/3b766adb-aca3-4cfe-87ea-590dadcc2c5e' -H 'Connection: keep-alive' -H 'X-Skygear-API-Key: tailorm-key' --data-binary '{"action":"record:save","api_key":"tailorm-key","access_token":"3e0add12-f662-4ae3-9936-c5737024816b","database_id":"_public","records":[{"_id":"TailormBooking/3b766adb-aca3-4cfe-87ea-590dadcc2c5e","_access":[{"public":true,"level":"write"}],"notes":""}]}' --compressed

PermissionDenied:
{"result":[{"_id":"TailormBooking/3b766adb-aca3-4cfe-87ea-590dadcc2c5e","_type":"error","code":102,"message":"no permission to modify","name":"PermissionDenied"}]}

Fix via d243718


fix: make all bash script pass shellcheck

All our bash scripts are now passing the shellcheck linter.
Signed-off-by: Sébastien Han<EMAIL_ADDRESS>

@leseb Is this PR related to #933?

Somehow but no direct connection. This PR cleans up all the warnings. The other one was partial and I think #933 could stick with the CI job only. Thanks!

Somehow but no direct connection. This PR cleans up all the warnings. The other one was partial and I think #933 could stick with the CI job only. Thanks!

My thinking is that on adding checker/linter is a follows:

Add checker to code base
Enable checker and fix all suggestions

When you merge 1 you then merge 2. This way you have a base where going forward the checker is in place, so any PR afterwards knows it is introducing the checker errors.
Will merging this PR just fix the shell checker errors for this moment in time? If so, what about the next person who makes changes/adds scripts?

Sorry for the confusion, my thinking is that this PR should be merged since it fixes all the shellcheck issues, then #933 should follow so that any changes to bash scripts will be lint.
Does that make sense? Thanks!
I commented on https://github.com/instructlab/instructlab/pull/933#issuecomment-2134972953 let's wait a few, if no response I'll just cherry-pick https://github.com/instructlab/instructlab/pull/933/commits/f1148a0edf169f9d12637869392fb1f89fadc618 in this PR.

Sorry for the confusion, my thinking is that this PR should be merged since it fixes all the shellcheck issues, then https://github.com/instructlab/instructlab/pull/933 should follow so that any changes to bash scripts will be lint.

If you do it this way, you are waiting on the tool to be added and enabled. My suggestions is to follow the approach in https://github.com/instructlab/instructlab/issues/77 where tools are added first and then PR follows with tool enabled in CI/CD containing all changes in relevant files as suggested by the tool. This way you are enabled and ready to go.

@leseb I see that #933 is closed and not being continued. Does this mean we run shellcheck on the fly and update changes like this PR? Or are we going to add it to CI/CD?

I'm writing the CI job at the moment and will push shortly an update to this PR.

Simulated failure can be seen here

Need https://github.com/instructlab/instructlab/pull/1246 for CI to be green

It looks like @hickeyma 's comments were addressed, so I'm going to drop that pending review and let this merge. Let me know if I missed anything, though!


Enhancement: Remove tokio::io::BufReader required in file reading

Motivation
In current, buffered reading is a default behavior, such as:
https://github.com/tonbo-io/tonbo/blob/40c23d38528456685f1e94d55b52ce6d589004fb/src/ondisk/scan.rs#L25-L26
and
https://github.com/tonbo-io/tonbo/blob/40c23d38528456685f1e94d55b52ce6d589004fb/src/ondisk/sstable.rs#L29-L30
however, some filesystem implementations actually does not need be buffered, such as io_uring based fs, remove it and make it required by specific fs implementation would be better.

Also, tokio::fs::AsyncFile already contained a buffer: https://docs.rs/tokio/latest/tokio/fs/struct.File.html#method.set_max_buf_size


feat: migrate Debian/RPM package from init to systemd

What type of PR is this?
/kind feature

Any specific area of the project related to this PR?
/area build

What this PR does / why we need it:
Current Linux Os migrates from init to systemd, so we update the Falco Debian/RPM packages to systemd also.
Which issue(s) this PR fixes:

Fixes #1265
Special notes for your reviewer:

tested on Ubuntu Focal
mkdir -p build
cd build
cmake ..
make package
ls *.deb | xargs dpkg -i
/usr/bin/falco-driver-loader
systemctl status falco.service
systemctl start falco.service
systemctl status falco.service

tested on CentOS 8 & openSUSE Leap 15.2
mkdir -p build
cd build
cmake ..
make package
ls *.rpm | xargs rpm -i
/usr/bin/falco-driver-loader
systemctl status falco.service
systemctl start falco.service
systemctl status falco.service

Does this PR introduce a user-facing change?:

update: Debian/RPM package migrated from init to systemd

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign leogr
You can assign the PR to them by writing /assign @leogr in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Welcome @jenting! It looks like this is your first PR to falcosecurity/falco 🎉

Thank you for your contribution!
Could we do the same thing for the DEB package too?

np! Let me make some code change and test on a Debian-based operating system.

Ready to be reviewed :rocket:

/milestone 0.27.0

I tried the systemd unit file from this on ubuntu and it worked well.
I've been using the juju4 ansible module which provides its own systemd unit file for several months. The diff is below:
0a1,3
> #
> # Ansible managed
> #
3c6
<
---
>
12a16
> PrivateTmp=true
23,27c27,31
< # PANIC: unprotected error in call to Lua API (runtime code generation failed, restricted kernel?)
< # https://www.freelists.org/post/luajit/luajit-crashes-with-grsec-kernel,1
< # MemoryDenyWriteExecute=true
< # PrivateMounts=true
<
---
> #   PANIC: unprotected error in call to Lua API (runtime code generation failed, restricted kernel?)
> #   ?https://www.freelists.org/post/luajit/luajit-crashes-with-grsec-kernel,1
> #MemoryDenyWriteExecute=true
> #PrivateMounts=true
>

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign leogr after the PR has been reviewed.
You can assign the PR to them by writing /assign @leogr in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@nibalizer really thanks to the testing and sharing on the falco works well with juju4 provided systemd service file works well for serverl months.
I've updated the PR to remove the comments in the systemd file, PTAL @leogr if you have time.

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign fntlnz after the PR has been reviewed.
You can assign the PR to them by writing /assign @fntlnz in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

This is ready to review.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: leogr
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [leogr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

LGTM label has been added.  Git tree hash: 4f76e27830f9bdbd8358c119cca9b6362c8519aa

/hold

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: fntlnz, leogr
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [fntlnz,leogr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/hold cancel

I believe it's now time to drop support for init.d based systems now.
Thanks for your patience on this @jenting and for pushing this forward for almost 1.5 years!
I'm signaling this as a breaking change in the release notes.

:clap:


Non-commercial open source project support

Hi,
I'm a maintainer of TinyUSB, the most widely used open-source cross-platform USB Host/Device stack for embedded system. It's also used by Espressif ESP-IDF and Raspberry Pi Pico SDK.
We'd like to integrate IAR into Github CI, I want to know what option we have :

Is IAR Build Tool offer license to non-commercial open source project ?
Can we use IAR EWARM with 32k evaluation license ? (So a Windows runner is needed)

Thanks,

Thanks for your support ! Now TinyUSB has integrated IAR CI :https://github.com/hathach/tinyusb/tree/support-iar

thank you for the opensource license, initial IAR support for stm32 (f0, f1 ,f3, f4, f7, g4, h7, l0) are already ported to build with IAR BX tools. https://github.com/hathach/tinyusb/pull/1876
There may be more ports in the future, but this is as good as it could get for initial PR.

Excellent news, @hathach!


Almost - please allow me to set a sync slicer as well.

This is a great development in the product and I've managed to figure out my url for filtering.
https://app.powerbi.com/groups/me/apps/....<content redacted>?filter=Date_x0020_Week_x0020_Ending~2FWeek_x0020_Ending_x0020_Date%20ge%202018-08-01%20and%20Date_x0020_Week_x0020_Ending~2FWeek_x0020_Ending_x0020_Date%20le%202018-08-04
It however doesn't quite work for me. I have a Synced filter/slicer on the landing page. What I really need to be able to do is set the value in that slicer as directed by the filter in the url. Then all would be well.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 70b360f3-21a7-adc2-896a-b1dfd5d9e384
Version Independent ID: 15e8c09f-46ad-68f9-75e6-c3d5c2c214e5
Content: Add Power BI report parameters using the url - Power BI
Content Source: powerbi-docs/service-url-filters.md
Service: powerbi
GitHub Login: @mihart
Microsoft Alias: mihart

Hi @dgwilson
I'm sorry, but there is no plan to allow you to set the filter state in URL parameters.  You could add it to ideas.powerbi.com and if it gets enough votes the Product team might implement it...
HTH,
Michele


[healthcareapis] Bump version to 2021-11-01

This checklist is used to make sure that common guidelines for a pull request are followed.

Bump version to 2021-11-01 stable.
az healthcareapis service Add new argument --login-servers to support adding login servers to the service instance.
az healthcareapis service Add new argument --oci-artifacts to support specifying open container initiative artifacts.
az healthcareapis private-endpoint-connection Deprecate argument --private-link-service-connection-state-actions-required since sdk version 2020-03-30
az healthcareapis private-endpoint-connection Deprecate argument --private-link-service-connection-state-description since sdk version 2020-03-30
az healthcareapis private-endpoint-connection Deprecate argument --private-link-service-connection-state-status since sdk version 2020-03-30
az healthcareapis private-endpoint-connection Add new argument --private-link-service-connection-state to support specifying information about the state of the connection between service consumer and provider.
Add new subgroups az healthcareapis workspace to Manage workspace with healthcareapis.
Add new subgroups az healthcareapis workspace dicom-service to Manage dicom service with healthcareapis.
Add new subgroups az healthcareapis workspace fhir-service to Manage fhir service with healthcareapis.
Add new subgroups az healthcareapis workspace iot-connector to Manage iot connector with healthcareapis.
Add new subgroups az healthcareapis workspace iot-connector fhir-destination to Manage iot connector fhir destination with healthcareapis.
Add new subgroups az healthcareapis workspace private-endpoint-connection to Manage workspace private endpoint connection with healthcareapis.
Add new subgroups az healthcareapis workspace private-link-resource to Manage workspace private link resource with healthcareapis.

For new extensions:

[ ] My extension description/summary conforms to the Extension Summary Guidelines.

About Extension Publish
There is a pipeline to automatically build, upload and publish extension wheels.
Once your pull request is merged into main branch, a new pull request will be created to update src/index.json automatically.
The precondition is to put your code inside this repository and upgrade the version in the pull request but do not modify src/index.json.

This PR will update healthcareapis to 2021-11-01.
And https://github.com/Azure/azure-cli/issues/21536 will closed by this PR.


Resolves #1528

Add input paragraph
https://github.com/MicrosoftDocs/windows-powershell-docs/issues/1528

Docs Build status updates of commit 15f924e:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docset/windows/hyper-v/Remove-VMHardDiskDrive.md
:white_check_mark:Succeeded

docset/winserver2012-ps/hyper-v/Remove-VMHardDiskDrive.md
:white_check_mark:Succeeded

docset/winserver2012r2-ps/hyper-v/Remove-VMHardDiskDrive.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

@dariomws306 @get-itips @tseward
Peer review needed for this PR. Thanks.


MDM_IP should be MDM_IPS

Check the variable, it should be MDM_IPS for Gateway installation

Hey, but this looks not be fixed in current code.

True...fixed


Enhance error logging

Prevent to fall into error logging recursion when it doesn't have
request info (Print 'Unknown' when the request is still None
Print repr(e) rather than str(e)

I don't know if I would be entirely comfortable giving the consumer of an API a full stack trace if the server is not in debug mode.

When I met the error in the write_error, it was unable to find where it is raised at all. (because it is already inside of the error handling code).
I agree verbose full trace can be uncomfortable for normal write_response case. How about changing to the full trace for write_error and no trace in write_response like before?

@youknowone is there a way to check if we are in debug mode and then only write the full trace in those cases?

Ok, I fonud bail_out is already handling the traceback case. Maybe I just needed to add 'Unknown' to solve my problem. Just wrapped errors with repr() to show their class name which also has important information as the error message.

It appears as though you have a pep8 error:
sanic/server.py:182:80: E501 line too long (80 > 79 characters)

@seemethere thanks, I updated it


Size mismatch, Error(s) in loading model finetuned by lora

是否已有关于该错误的issue或讨论？ | Is there an existing issue / discussion for this?

[X] 我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions

该问题是否在FAQ中有解答？ | Is there an existing answer for this in FAQ?

[X] 我已经搜索过FAQ | I have searched FAQ

当前行为 | Current Behavior
I finetune the QWen-7B, and when I use the finetuned model, I met some error:
root@1fc7d6985d8b:/Fine/Qwen-main# python3 cli_demo.py
/usr/local/lib/python3.8/dist-packages/transformers/utils/generic.py:260: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  torch.utils._pytree._register_pytree_node(
The model is automatically converting to bf16 for faster inference. If you want to disable the automatic precision, please manually add bf16/fp16/fp32=True to "AutoModelForCausalLM.from_pretrained".
Try importing flash-attention for faster inference...
Warning: import flash_attn rotary fail, please install FlashAttention rotary to get higher efficiency https://github.com/Dao-AILab/flash-attention/tree/main/csrc/rotary
Warning: import flash_attn rms_norm fail, please install FlashAttention layer_norm to get higher efficiency https://github.com/Dao-AILab/flash-attention/tree/main/csrc/layer_norm
Warning: import flash_attn fail, please install FlashAttention to get higher efficiency https://github.com/Dao-AILab/flash-attention
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:10<00:00,  1.30s/it]
You are resizing the embedding layer without providing a `pad_to_multiple_of` parameter. This means that the new embeding dimension will be 151851. This might induce some performance reduction as *Tensor Cores* will not be available. For more details  about this, or help on choosing the correct value for resizing, refer to this guide: https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc
Traceback (most recent call last):
  File "cli_demo.py", line 217, in <module>
    main()
  File "cli_demo.py", line 123, in main
    model, tokenizer, config = _load_model_tokenizer(args)
  File "cli_demo.py", line 60, in _load_model_tokenizer
    model = AutoPeftModelForCausalLM.from_pretrained(
  File "/usr/local/lib/python3.8/dist-packages/peft/auto.py", line 128, in from_pretrained
    return cls._target_peft_class.from_pretrained(
  File "/usr/local/lib/python3.8/dist-packages/peft/peft_model.py", line 353, in from_pretrained
    model.load_adapter(model_id, adapter_name, is_trainable=is_trainable, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/peft/peft_model.py", line 697, in load_adapter
    load_result = set_peft_model_state_dict(self, adapters_weights, adapter_name=adapter_name)
  File "/usr/local/lib/python3.8/dist-packages/peft/utils/save_and_load.py", line 249, in set_peft_model_state_dict
    load_result = model.load_state_dict(peft_model_state_dict, strict=False)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 2153, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for PeftModelForCausalLM:
        size mismatch for base_model.model.transformer.wte.modules_to_save.default.weight: copying a param with shape torch.Size([151936, 4096]) from checkpoint, the shape in current model is torch.Size([151851, 4096]).
        size mismatch for base_model.model.lm_head.modules_to_save.default.weight: copying a param with shape torch.Size([151936, 4096]) from checkpoint, the shape in current model is torch.Size([151851, 4096]).

I finetuned the model by use:
bash finetune_lora_single_gpu.sh -d my_train_data.json

and I modified my cli-demo.py follow the tutorial:
    model = AutoPeftModelForCausalLM.from_pretrained(
        model_path, # path to the output directory or model name
        device_map=device_map,
        trust_remote_code=True,
    ).eval()

I found some probably related issues like #419  #482 , but they can't solve my problem.
期望行为 | Expected Behavior
demo runs normally.
复现方法 | Steps To Reproduce
No response
运行环境 | Environment
- OS: 22.04.1-Ubuntu
- Python: 3.8
- Transformers:4.32.0
- PyTorch:2.2.1
- CUDA 12.1

备注 | Anything else?
No response

Something seems wrong with the vocab_size (which is the size of the embedding, not the actual vocabulary size) in config.json and the pad_to_multiple_of setting.
Please try upgrade transformers<4.38.0 and downgrade peft<0.8.0 first and provide the content of config.json.

Okay, I upgrade transformers to 4.38.0, and use peft=0.7.0 now, but meet some errors which I haven't met when I use peft=0.9.0.

config.json is here, I copied it from https://huggingface.co/Qwen/Qwen-7B-Chat/blob/main/config.json
{
  "architectures": [
    "QWenLMHeadModel"
  ],
  "auto_map": {
    "AutoConfig": "configuration_qwen.QWenConfig",
    "AutoModelForCausalLM": "modeling_qwen.QWenLMHeadModel"
  },
  "attn_dropout_prob": 0.0,
  "bf16": false,
  "emb_dropout_prob": 0.0,
  "fp16": false,
  "fp32": false,
  "hidden_size": 4096,
  "intermediate_size": 22016,
  "initializer_range": 0.02,
  "kv_channels": 128,
  "layer_norm_epsilon": 1e-06,
  "max_position_embeddings": 32768,
  "model_type": "qwen",
  "no_bias": true,
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "onnx_safe": null,
  "rotary_emb_base": 10000,
  "rotary_pct": 1.0,
  "scale_attn_weights": true,
  "seq_length": 8192,
  "tie_word_embeddings": false,
  "tokenizer_class": "QWenTokenizer",
  "transformers_version": "4.32.0",
  "use_cache": true,
  "use_dynamic_ntk": true,
  "use_flash_attn": "auto",
  "use_logn_attn": true,
  "vocab_size": 151936
}

There should be an adapter_config.json as well. Let's see what's there. I think peft is changing the vocab_size somewhere.

Okay, I upgrade transformers to 4.38.0, and use peft=0.7.0 now, but meet some errors which I haven't met when I use peft=0.9.0.
config.json is here, I copied it from https://huggingface.co/Qwen/Qwen-7B-Chat/blob/main/config.json
{
  "architectures": [
    "QWenLMHeadModel"
  ],
  "auto_map": {
    "AutoConfig": "configuration_qwen.QWenConfig",
    "AutoModelForCausalLM": "modeling_qwen.QWenLMHeadModel"
  },
  "attn_dropout_prob": 0.0,
  "bf16": false,
  "emb_dropout_prob": 0.0,
  "fp16": false,
  "fp32": false,
  "hidden_size": 4096,
  "intermediate_size": 22016,
  "initializer_range": 0.02,
  "kv_channels": 128,
  "layer_norm_epsilon": 1e-06,
  "max_position_embeddings": 32768,
  "model_type": "qwen",
  "no_bias": true,
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "onnx_safe": null,
  "rotary_emb_base": 10000,
  "rotary_pct": 1.0,
  "scale_attn_weights": true,
  "seq_length": 8192,
  "tie_word_embeddings": false,
  "tokenizer_class": "QWenTokenizer",
  "transformers_version": "4.32.0",
  "use_cache": true,
  "use_dynamic_ntk": true,
  "use_flash_attn": "auto",
  "use_logn_attn": true,
  "vocab_size": 151936
}

Hi, have you solved this problem?

Okay, I upgrade transformers to 4.38.0, and use peft=0.7.0 now, but meet some errors which I haven't met when I use peft=0.9.0.
config.json is here, I copied it from https://huggingface.co/Qwen/Qwen-7B-Chat/blob/main/config.json
{
  "architectures": [
    "QWenLMHeadModel"
  ],
  "auto_map": {
    "AutoConfig": "configuration_qwen.QWenConfig",
    "AutoModelForCausalLM": "modeling_qwen.QWenLMHeadModel"
  },
  "attn_dropout_prob": 0.0,
  "bf16": false,
  "emb_dropout_prob": 0.0,
  "fp16": false,
  "fp32": false,
  "hidden_size": 4096,
  "intermediate_size": 22016,
  "initializer_range": 0.02,
  "kv_channels": 128,
  "layer_norm_epsilon": 1e-06,
  "max_position_embeddings": 32768,
  "model_type": "qwen",
  "no_bias": true,
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "onnx_safe": null,
  "rotary_emb_base": 10000,
  "rotary_pct": 1.0,
  "scale_attn_weights": true,
  "seq_length": 8192,
  "tie_word_embeddings": false,
  "tokenizer_class": "QWenTokenizer",
  "transformers_version": "4.32.0",
  "use_cache": true,
  "use_dynamic_ntk": true,
  "use_flash_attn": "auto",
  "use_logn_attn": true,
  "vocab_size": 151936
}

Hi, have you solved this problem?

I'm sorry I didn't follow up on this issue.


Add binaryscraping/swift-log-supabase to README

Adds swift-log-supabase as a compatible library.
A backend that sends logs to Supabase.
Motivation:
Supabase is an open source alternative to Firebase and it's being used a lot recently.

@swift-server-bot test this please


Add interface for partial dataset loading

Proposed changes:
Basic interface in order to tackle the remaining issues for partial dataset loading.
closes #7270
Status (please check what you already did):

[ ] added some tests for the functionality
[ ] updated the documentation
[ ] updated the changelog (please check changelog for instructions)
[x] reformat files using black (please check Readme for instructions)

@dakshvar22 Can you briefly take a look? It would be great to get this merged so that we can start working on the other issues :) thanks.


Unable to run storybook from source

Describe the bug
As per demo run
Development
Run storybook
To view the storybook locally, you should:
Clone superset-ui-plugins repo.
Run yarn install && yarn build in the superset-ui-plugins root directory.
Change to the demo directory cd packages/superset-ui-plugins-demo.
Run yarn run storybook. This will open up a dev server at http://localhost:9001.
Expected behavior
Start superset-ui-plugins-demo plugin in storyboard
Screenshots

Please help me how to run

Sorry for wrong information.Its not an issue,its just an configuration.
Thanks,problem has been solved


[issue #1] convert f string

Description
https://github.com/devrel-kr/azure-sdk-for-python/issues/1 이 이슈를 해결시켰어요
All SDK Contribution checklist:

[x] The pull request does not introduce [breaking changes]
[x] CHANGELOG is updated for new features, bug fixes or other significant changes.
[x] I have read the contribution guidelines.

General Guidelines and Best Practices

[x] Title of the pull request is clear and informative.
[x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[ ] Pull request includes test coverage for the included changes.

test


Consider modifgying advice for 1.4.13 Content on Hover and Focus

https://appt.org/en/guidelines/wcag/success-criterion-1-4-13
Suggested improvements
The text refers to content that pops up when elements are touched. This is not really what the SC covers, though - it covers mouse hover and keyboard focus receipt.
The solution of adding a close button would not work snce ot would mean users have to move focus to activate it.
The SC applies when apps are used with attached mice or keyboard (even though hover and focus will be rare in apps for obvious reasons).

Hi @detlevhfischer thanks for your suggestion. I'll discuss it with other project members in April, when I'm back from my holiday.

Hi @detlevhfischer , like you mentioned, pointer hover is something that doesn't really happen in apps. Keyboard focus is probably more common. Long press might be most equivalent to 'pointer hover' as intended in the SC. Perhaps we can rename touch/click to 'long press' and/or keyboard focus. What do you think?

Hi @detlevhfischer what do you think of the text suggested on April 11?

@JJdeGroot Hover is definitely an option when a USB or bluetooth mouse is attached to a tablet, even though hover events it will be rarely supported since most users do use touch to interact with the app. But mobile OS become more prevalent and contain more complex applications where the use of keyboard or mouse might make sense. So it is not clear that hover can simply be discounted.
As to your long press idea: There simply are as yet no particular requirements for long press contained in 1.4.13 - so if you include requirements for that, it should be clear that this is not a normative requirement from WCAG 2.X or even the current update of WCAG2ICT, but something you consider worth including on its own merit.

Hi @detlevhfischer  I have changed the sentence to:
English:
Ensure it is possible to hide information which appears automatically after hovering with the pointer or focusing with assistive technologies.
Dutch:
Zorg dat het mogelijk is om informatie te verbergen indien deze automatisch verschijnt na hover met de aanwijzer of focus met hulpmiddelen.
This sentence is now more in line with the official WCAG definition.

@JJdeGroot I think apart from hovering (pointer) focusing should not be restricted to "focusing with assistive technologies". The main case for th ekeyboard focus half of 1.4.13 will be checking keyboard operation of an app by coupling it with an external keyboard, focusing elements with tab (or arrow keys as it may be), and checking that any content that pops up on kb focus can be closed (e.g via ESC). The focus of 1.4.13 is clearly on sighted users, not screenreader users. I agree that content appearing when focusing elements in read mode (e.g. via touch swipe gestures) should probably also meet the requirements, but since WCAG does not spell out if the same requirement applies to content when a screen reader is active, I would avoid any explicit reference to assisitve technology use here.

Hi @detlevhfischer good points, I have changed assistive technologies to keyboard. Also updated some more phrases where assistive technologies were mentioned.


i18nRouter overwriting original NextRequest headers and breaks Next 13 caching

Next: ^14.0.2
next-i18n-router: ^4.1.2
I am not super knowledgeable when it comes to headers and middleware but I have been having a caching issue in my Next 14 project for the last few months that is causing everything to be revalidated on request (default behavior is to never revalidate). I haven't been able to narrow the issue down until now when I noticed that a TON of headers were lost when turning my NextRequest into an i18nRequest.
I don't know if this is a bug or an intentional behavior for some reason or how I can make it not purge my headers?
middleware.ts
export function middleware(request: NextRequest) {
  console.log("--- Original Headers ---");
  request.headers.forEach((value, key) => {
    console.log(key + ": " + value);
  });

  let i18nRequest = i18nRouter(request, {
    locales: Object.keys(locales),
    defaultLocale: "en",
    routingStrategy: "dynamicSegment",
  });

  // Get the device type
  const { device } = userAgent(request);

  i18nRequest.headers.set(
    "x-device-type",
    device.type === "mobile" ? "mobile" : "desktop"
  );

  console.log("--- i18n Headers ---");
  i18nRequest.headers.forEach((value, key) => {
    console.log(key + ": " + value);
  });

  return i18nRequest;
}

Here are the headers logged prior to building the i18nRequest
--- Original Headers ---
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
cache-control: max-age=0
connection: keep-alive
host: localhost:3000
sec-ch-ua: "Not_A Brand";v="8", "Chromium";v="120", "Google Chrome";v="120"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: same-origin
sec-fetch-user: ?1
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36
x-forwarded-for: ::1
x-forwarded-host: localhost:3000
x-forwarded-port: 3000
x-forwarded-proto: http

And here are the headers after passing through i18nRouter
--- i18n Headers ---
x-device-type: desktop
x-middleware-rewrite: http://localhost:3000/en/
x-next-i18n-router-locale: en

Probably related https://github.com/i18nexus/next-i18n-router/issues/29

I think there might be confusion about "request" vs "response". The i18nRouter function generates a response, not a request. So to avoid confusion, I would rename your i18nRequest variable to i18nResponse.
i18nRouter generates a response with NextResponse.next(). This response is completely new. So it will not contain any headers (besides a "x-next-i18n-router-locale" header that is added). If you want to add other headers to your response, you can do so using i18nResponse.headers.set like you did at the bottom of your function with "x-device-type".
However
I notice in the docs that the NextResponse.next() can take a request argument to forward request headers onto the response: https://nextjs.org/docs/app/api-reference/functions/next-response#next
I think this library should probably be doing this. It's beginning to sound like this is the default behavior of Next when there is no middleware involved.
We're currently looking into this. It is looking like we will likely be implementing forwarding request headers and will be released in the next version. Thanks for the report! Will keep you updated.
In the meantime, you may just want to manually add the request headers onto the i18nResponse.

Luckily this was an easy fix. Request headers are now forwarded onto the response in v5.1.0


Models refactor proposal

TBD

This PR was the foundation for other smaller ones where we applied the changes here proposed.


feat: add url in notarise schema

:tada: This PR is included in version 1.6.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 1.6.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Updates Roadmap

This PR adds some initial items to the Roadmap and introduces some loose timeboxing.
Mainly I'd like this PR to serve as a starter for a discussion as to what should be in our Roadmap as well as what should be the process for updating it and what the Roadmap should be used for.
The previous version did correctly indicate the kind of things we are interested in, but in practice it was never used, so I think perhaps a more concrete Roadmap could work better.
Suggestions for ideas for process, structure and other items are very welcome- I expect that the actual contents will liikely be very different from the original commit.
For inspiration as to how other project do it @wallrj pointed us at Github's Roadmap https://github.com/orgs/github/projects/4247/views/1
NONE

Signed-off-by: irbekrm<EMAIL_ADDRESS>

Thanks @irbekrm !
/assign

/test pull-cert-manager-make-test

/override pull-cert-manager-deps
(That test was removed and prow doesn't seem to gracefully handle a test being removed for open PRs)

Updates as per dev meeting

/test pull-cert-manager-e2e-v1-24


Add GitHub data connector; add Markdown partitioner

Closes #256
Hello!
Pull Request overview

Add GitHub data connector for ingest.

I have ticked off the checklist.
The connector can process a single repository and recursively load all documents in the repo.
Files with extensions that are not supported by partition.auto are skipped.
Supports downloading from a specific git branch.
Supports the ability to filter documents through globs.

Add partition_md to support partitioning Markdown files.

Implemented using python-markdown, which internally converts the markdown into HTML. This is perhaps not ideal, but it works well enough for now.
Add FileType.MD and support for partition_md in auto.partition.

The data connector checklist

[x] Create a new module under unstructured/ingest/connector/ implementing the 3 abstract base classes, similar to unstructured/ingest/connector/s3_connector.py.

[x] The subclass of BaseIngestDoc overrides process_file() if extra processing logic is needed other than what is provided by auto.partition().

[x] Update unstructured/ingest/main.py with support for the new connector.
[x] Create a folder under examples/ingest that includes at least one well documented script.
[x] Add a script test_unstructured_ingest/test-ingest-<the-new-data-source>.sh. It's json output files should have a total of no more than 100K.
[x] Git add the expected outputs under test_unstructured_ingest/expected-structured-output/<folder-name-relevant-to-your-dataset> so the above test passes in CI.
[x] Add a line to test_unstructured_ingest/test-ingest.sh invoking the new test script.
[x] If additional python dependencies are needed for the new connector:

[x] Add them as an extra to setup.py.
[x] Update the Makefile, adding a target for install-ingest-<name> and adding another pip-compile line to the pip-compile make target. See this commit for a reference.
[x] The added dependencies should be imported at runtime when the new connector is invoked, rather than as top-level imports.

[x] Honors the conventions of BaseConnectorConfig defined in unstructured/ingest/interfaces.py which is passed through the CLI:

[x] If running with an .output_dir where structured outputs already exists for a given file, the file content is not re-downloaded from the data source nor is it reprocessed. This is made possible by implementing the call to MyIngestDoc.has_output() which is invoked in MainProcess._filter_docs_with_outputs.
[x] Unless .reprocess is True, then documents are always reprocessed.
[x] If .preserve_download is True, documents downloaded to .download_dir are not removed after processing.
[x] Else if .preserve_download is False, documents downloaded to .download_dir are removed after they are successfully processed during the invocation of MyIngestDoc.cleanup_file() in process_document
[x] Does not re-download documents to .download_dir if .re_download is False, enforced in MyIngestDoc.get_file()
[x] Prints more details if .verbose similar to unstructured/ingest/connector/s3_connector.py.

Details
Installation
I've added pygithub==1.57.0 as a dependency, installable via pip install unstructured["github"]. Note that the newer 1.58.0 does not work due to this issue: https://github.com/PyGithub/PyGithub/issues/2436, it prevents us from using a Repository instance in a multiprocessing loop.
I've also added markdown as a core dependency, required for partition_md, and types-Markdown as a test requirement to satisfy mypy. I re-ran pip-compile to update the requirements files afterwards.
CLI Help
The usage of the unstructured-ingest CLI is now like so:
Usage: unstructured-ingest [OPTIONS]

Options:
  --s3-url TEXT                   Prefix of s3 objects (files) to download.
                                  E.g. s3://bucket1/path/. This value may also
                                  be a single file.
  --s3-anonymous                  Connect to s3 without local AWS credentials.
  --github-url TEXT               URL to GitHub repository, e.g.
                                  "https://github.com/Unstructured-
                                  IO/unstructured", or a repository owner/name
                                  pair, e.g. "Unstructured-IO/unstructured"
  --github-access-token TEXT      A GitHub access token, see https://docs.gith
                                  ub.com/en/authentication/keeping-your-
                                  account-and-data-secure/creating-a-personal-
                                  access-token
  --github-branch TEXT            The branch for which to fetch files from. If
                                  not given, the default repository branch is
                                  used.
  --github-file-glob TEXT         A comma-separated list of file globs to
                                  limit which types of files are accepted,
                                  e.g. '*.html,*.txt'
  --re-download / --no-re-download
                                  Re-download files from s3 even if they are
                                  already present in --download-dir.
  --download-dir TEXT             Where s3 files are downloaded to, defaults
                                  to tmp-ingest-<6 random chars>.
  --preserve-downloads            Preserve downloaded s3 files. Otherwise each
                                  file is removed after being processed
                                  successfully.
  --structured-output-dir TEXT    Where to place structured output .json
                                  files.
  --reprocess                     Reprocess a downloaded file from s3 even if
                                  the relevant structured output .json file in
                                  --structured-output-dir already exists.
  --num-processes INTEGER         Number of parallel processes to process docs
                                  in.  [default: 2]
  -v, --verbose
  --help                          Show this message and exit.

Note the four new github-related arguments. In particular, I want to point out that for --github-url, both https://github.com/Unstructured-IO/unstructured and Unstructured-IO/unstructured are valid. Beyond that, --github-file-glob can be used to select specific data, e.g. with "*.html,*.txt,*.md".
Sample usage
An example usage would be the following:
unstructured-ingest \
    --github-url Unstructured-IO/unstructured \
    --github-branch main \
    --structured-output-dir github-ingest-output \
    --num-processes 2 \
    --verbose

Markdown partitioner
This PR does not only include a GitHub connector, but also a partitioner for markdown. I rely on python-markdown, which internally converts the markdown into HTML text, after which I call partition_html. The new partition_md has this signature:
def partition_md(
    filename: Optional[str] = None,
    file: Optional[IO] = None,
    text: Optional[str] = None,
    url: Optional[str] = None,
    include_page_breaks: bool = False,
    include_metadata: bool = True,
    parser: VALID_PARSERS = None,
) -> List[Element]:

I want to point out one caveat of this approach. Normally, when calling a partition method with filename and include_metadata=True, then the filename will be included in the output. However, my implementation always calls partition_html via text=..., even if partition_md is called with a filename. In short, the filename is not included in outputs by partition_md.
For the time being I think this is a minor issue - it is already miles better than not having any Markdown support, after all. I also wanted to keep the focus of this PR on the GitHub connector.
I also added some very simple tests to the markdown partitioner based on the HTML partitioner.
Markdown sample output
See here a snippet from my output when running:
unstructured-ingest --github-url Unstructured-IO/unstructured --github-file-glob 'Readme.md' --structured-output-dir github-unstructured-readme-output --verbose

With other words, I downloaded from this repository, using the glob Readme.md, which means I only fetch the README. The output folder contained only README.md.json containing this output. As you can see, the titles are correctly discovered etc.
...
  {
    "element_id": "6fc70513a3165acfe9986eda09da3e36",
    "text": "Open-Source Pre-Processing Tools for Unstructured Data",
    "type": "Title",
    "metadata": {
      "page_number": 1
    }
  },
  {
    "element_id": "dec0569ba095e8dfad354bbdd6a624c1",
    "text": "The unstructured library provides open-source components for pre-processing text documents\nsuch as PDFs, HTML and Word Documents. These components are packaged as bricks ð§±, which provide\nusers the building blocks they need to build pipelines targeted at the documents they care\nabout. Bricks in the library fall into three categories:",
    "type": "UncategorizedText",
    "metadata": {
      "page_number": 1
    }
  },
  {
    "element_id": "de32231a1dac79f3e9baea4e970ad305",
    "text": ":jigsaw: Partitioning bricks that break raw documents down into standard, structured\n  elements.",
    "type": "UncategorizedText",
    "metadata": {
      "page_number": 1
    }
  },
  {
    "element_id": "3c3ad9bc65fd3100568ecad1dbd768e5",
    "text": ":broom: Cleaning bricks that remove unwanted text from documents, such as boilerplate and\n  sentence\n  fragments.",
    "type": "UncategorizedText",
    "metadata": {
      "page_number": 1
    }
  },
  {
    "element_id": "6388255e5f0ea0819eed2c8b3b3bd02f",
    "text": ":performing_arts: Staging bricks that format data for downstream tasks, such as ML inference\n  and data labeling.",
    "type": "UncategorizedText",
    "metadata": {
      "page_number": 1
    }
  },
  {
    "element_id": "8329a04e799e36105f29583b4f10814c",
    "text": ":eight_pointed_black_star: Quick Start",
    "type": "Title",
    "metadata": {
      "page_number": 1
    }
  },
  {
    "element_id": "115d9086aac6be868679eb07e418e65d",
    "text": "Use the following instructions to get up and running with unstructured and test your\ninstallation.",
    "type": "UncategorizedText",
    "metadata": {
      "page_number": 1
    }
  },
...

Let me know if you need anything of me at this point! I hope this will be helpful to you all!

Tom Aarsen

Some additional, unrelated comments:

reprocess is an argument on the BaseConnectorConfig, but it doesn't have to be there: It is only used on MainProcess instead:
https://github.com/Unstructured-IO/unstructured/blob/e61ce2cc0065b4afb2af89df648058909ffd4f74/unstructured/ingest/interfaces.py#L50
The initialize method on the BaseConnector never gets called:
https://github.com/Unstructured-IO/unstructured/blob/e61ce2cc0065b4afb2af89df648058909ffd4f74/unstructured/ingest/interfaces.py#L25-L29
That also goes with the initialize method of the S3Connector, which actually has code in it. I didn't test thoroughly, but I don't think these lines are ever called:
https://github.com/Unstructured-IO/unstructured/blob/e61ce2cc0065b4afb2af89df648058909ffd4f74/unstructured/ingest/connector/s3_connector.py#L166-L173

Perhaps I can make a PR to address these changes soon.

Tom Aarsen

@tomaarsen , First off, lots of great stuff in this PR!
I did some testing on the markdown/detection partitioning, and hit an issue, so the rest of this comment is focused on that.
At least in my mac env, the filetype for .md files is resolving as FileType.TXT in unstructured/partition/auto.py, so it never gets into the clause:
    elif filetype == FileType.MD:

I was initially looking for diff's with this script between main and this branch, and noticed the outputs were the same.
$ cat ~/tmp/dump-elems.py
import sys
from unstructured.partition.auto import partition
elems = partition(sys.argv[1])
output = "\n".join([f"{e.to_dict()['type']:20}  {e.text:<80}" for e in elems])
print(output)

I think the issue itself may have been misleading with markdown comment in that the text parser is actually ok-ish -- sorry about that!
For the sake of this PR, it is OK to punt on better markdown processing but also OK to try to improve it :)
FWIW, source code files also come through as FileType.TXT, which ends up with the text in multiple Elements even though each individual element is particularly meaningful.

Regarding:

Some additional, unrelated comments:

reprocess is an argument on the BaseConnectorConfig, but it doesn't have to be there: It is only used on MainProcess instead:

Yes, you're right, should be removed.

The initialize method on the BaseConnector never gets called

Good catch! I pushed the quick fix here:
https://github.com/Unstructured-IO/unstructured/pull/285

@tomaarsen , First off, lots of great stuff in this PR!
I did some testing on the markdown/detection partitioning, and hit an issue, so the rest of this comment is focused on that.
At least in my mac env (with libmagic installed, which is preferred), the filetype for .md files is resolving as FileType.TXT in unstructured/partition/auto.py, so it never gets into the clause:

Quick comment - I'll reply in more depth later. I think this is because I had accidentally installed both python-magic and python-magic-bin. The first of which breaks the magic import on Windows and sets a conditional to false such that it uses a backup system for determining the filetype: a mapping from ".md" to FileType.MD.

@cragwolfe I'd love to add proper support for .md files: it seems like a valuable addition to the product beyond the GitHub data connector itself. However, the current implementation of detect_filetype is unable to detect MD files correctly. I'll try to work around this!

I've made the markdown detection more robust. Could you perhaps try again to see if it works as intended on your device now?
I appreciate it.

I think it is important to establish the pattern of parsing markdown files explicitly, as this PR does -- thanks!
I think there is room for improvement in the markdown parser -- we should be able to use markdown native "types" rather than convert to html. But not a blocker for this PR and can be tracked in a separate issue.
exploration

I think there is room for improvement in the markdown parser -- we should be able to use markdown native "types" rather than convert to html. But not a blocker for this PR and can be tracked in a separate issue.

I agree, for the time being this "Markdown-via-HTML" support will be preferable to no markdown support or using the partition_text for it.
Alternatively, another PR can take it in the opposite direction and move to using pandoc as suggested in #215 for converting various different markup formats to e.g. html. This would theoretically add support for all filetypes that pandoc supports, although then the filetype detector would also have to be able to detect those other markup files correctly.
Perhaps an interesting discussion for later.

I'm seeing some errors that seem specific to github.py when processing the Unstructured repo:
bash -x examples/ingest/github/ingest.sh

fetching GitHubIngestDoc(repo=Repository(full_name="Unstructured-IO/unstructured"), path='examples/argilla-summarization/img/isw.png') - PID: 99063
ERROR:root:Failed to process GitHubIngestDoc(repo=Repository(full_name="Unstructured-IO/unstructured"), path='examples/argilla-summarization/img/isw.png')
Traceback (most recent call last):
  File "/Users/cragwolfe/r/unstructured/unstructured/ingest/doc_processor/generalized.py", line 23, in process_document
    doc.get_file()
  File "/Users/cragwolfe/r/unstructured/unstructured/ingest/connector/github.py", line 94, in get_file
    f.write(content_file.decoded_content)  # type: ignore
  File "/Users/cragwolfe/.pyenv/versions/unst/lib/python3.8/site-packages/github/ContentFile.py", line 59, in decoded_content
    assert self.encoding == "base64", f"unsupported encoding: {self.encoding}"
AssertionError: unsupported encoding: none

But this file may currently be processed with:
$ PYTHONPATH=. python -c "from unstructured.partition.auto import partition;
print('len of elems is '+str(len(partition('examples/argilla-summarization/img/isw.png'))))
len of elems is 6

Other docs this is an issue for: example-docs/fake-email-header.eml, example-docs/factbook.xml, example-docs/example-10k.html
Once these are clean, this PR should be good to go!

I've had a look into this, and it seems to be the result of poor documentation on the PyGitHub side. On the official GitHub documentation I could find some more information about this: https://docs.github.com/en/rest/repos/contents?apiVersion=2022-11-28#size-limits
As it turns out, files larger than 1MB are returned to have no contents and an encoding of "none". This caused an AssertionError on the decoded_content property, which expects to always see a "base64" encoding. I would have expected this to be documented on the PyGitHub side, or for a more useful error to be thrown.
Regardless, I can use the raw.githubusercontent link, e.g.:
https://raw.githubusercontent.com/Unstructured-IO/unstructured/main/unstructured/nlp/english-words.txt
As for the .eml file etc., it seems to consider that one a message/rfc822. I'm not sure why it does that - perhaps something with the file encoding?


Updating IP binding from local loop to accepting all traffic

I was using this example to setup a local server and found that this example uses a localhost loop IP. This creates an issue when I try to access this server from remote. Updating it to <IP_ADDRESS> will make it accept remote traffic as well which is probably what most people want.

realized this is not the actual package. I will leave this open in case you want this. But I will publish a PR in that package as well.


tgenv-version-name: [ERROR] Version could not be resolved

❯ terragrunt plan                                                                                ─╯
cat: /usr/local/Cellar/tgenv/0.0.3/version: No such file or directory
tgenv: tgenv-version-name: [ERROR] Version could not be resolved (set by /usr/local/Cellar/tgenv/0.0.3/version or tgenv use <version>)
/usr/local/Cellar/tgenv/0.0.3/libexec/tgenv-exec: line 22: /usr/local/Cellar/tgenv/0.0.3/versions//terragrunt: No such file or directory

My installation was done via Homebrew
❯ brew info tgenv                                                                                ─╯
==> tgenv: stable 0.0.3 (bottled), HEAD
Terragrunt version manager inspired by tfenv
https://github.com/cunymatthieu/tgenv
Conflicts with:
  terragrunt (because tgenv symlinks terragrunt binaries)
/usr/local/Cellar/tgenv/0.0.3 (19 files, 33.5MB) *
  Poured from bottle on 2022-11-21 at 16:07:21
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/tgenv.rb
License: MIT
==> Options
--HEAD
	Install HEAD version
==> Analytics
install: 426 (30 days), 1,252 (90 days), 5,361 (365 days)
install-on-request: 425 (30 days), 1,251 (90 days), 5,351 (365 days)
build-error: 0 (30 days)

The variable Shell it was exported: TGENV_AUTO_INSTALL=true
Has any Mac user experienced this issue and managed to resolve it?

It was solved, probably was a intermittence at mirror.


Plugin name refactor & set Gutenberg block category for design system.

Rename plugin code and constants to match github name (and make everything consistent so category works more cleanly)
Assign new category (CA Design System) to all blocks and patterns.
Bump CA Design System category to top of list in GB block picker.

Set plugin version to 1.0.1.

With the blocks loaded & bumped up to the top of the interface (when you click the default blue + icon), all our blocks are grouped in a category, so we can find them (and also speed up development and testing.)

@aaronhans  @wrendo  @kkoryaka @m-sullivan7  @adam-little
A preview is auto generated from the plugin code - this might be what "Example" is, and we will want to provide sensible defaults for our blocks. This will help the UX & onboarding content editors.
We can look into ordering the elements. Based on what I can see from the Gutenberg Blocks API, there is a lot of potential tailoring that can happen (over time) to improve the interface.

Right now they are alphabetical.

Wow, this is really cool! Great work Chach!

Did you already review the existing core WordPress twitter and button components? I hoped those would be good enough for 7/1.


can we stop to create file when they are no values

Hello ,
i use v5.3.0
I use JdbcRecordReader and writer.
I want write in my file if only my query return at least one line.
How can I do it please ?

On spring batch it s writer.setShouldDeleteIfEmpty(true);
How do it on EasyBatch  ?

You can't do it with the FileRecordWriter. It is not the responsibility of the writer to delete or move the file if no records have been written. You can do that using a JobListener#afterJob instead, something like:
public void afterJob(JobReport jobReport) {
    if (jobReport.getMetrics().getWriteCount() == 0) {
        delete(yourFile);
    }
}

Thank you !


Warn if LifeCycleManager is in use

I think about removing the LifeCycle support from Embulk. That feature is never used as far is I know.
Before removing that feature, trying to message it to users to expect users report to us when they're using it.
@kamatama41 @sakama Can you have a look?

Thanks! Merging... JFYI, the removal will be done like #1078.


Channel still stuck in force close, not releasing funds

Referring to Issue #6804
Channel has been stuck in pending_force_closing_channels for a few days and still has not released the funds. No longer sure what the solution is here.

If you do lncli pendingchannels do you see a value there for the amount of blocks remaining? If that isn't zero, then teh funds are still time locked.

"blocks_til_maturity": 264
is this what you mean?

It means that it was force closed by you and you can not make use of the funds until that number of blocks. The other party can challenge the force close if you did not use the latest state (tried to steal).

Ahh I see, that's good to know. In that case I'll close the issue.
Thanks for the help, I'm still new to all this so still trying to figure it out :)


Fix admin project detail prohibit rollup

Noticed the admin project details "Prohibit Project Name" collapse button wasn't working. This just adds the container <div class="box-body"> to fix it.
Review with whitespace off.

Thanks!


Self-hosted CPU Regression Tests

Proposed changes:

Enable regression tests on self-hosted not just for GPU but also for CPU

Status (please check what you already did):

[ ] added some tests for the functionality
[ ] updated the documentation
[ ] updated the changelog (please check changelog for instructions)
[ ] reformat files using black (please check Readme for instructions)

/modeltest
include:
 - dataset: ["financial-demo"]
   config: ["Sparse + BERT + DIET(seq) + ResponseSelector(t2t)"]


docs: add instructions for Search API Pantheon module (for Drupal)

The Issue
How This PR Solves The Issue
Manual Testing Instructions
Automated Testing Overview

Related Issue Link(s)
Release/Deployment Notes

All I am doing is summarizing what I shared on Slack. I have a lot of other things to do. Anyone who wants to take this further is welcome to.

As far as I can tell this isn't something that belongs in the DDEV docs. I think it also is a very obscure case where you're using the upstream Pantheon service with local development, right?   It's great that you sorted it out, but it would be better in Stack Overflow I think. Could you post it there? Do it as a question-and-answer where you click the checkbox to answer your own question. Thanks!


"true" must be quoted in docker compose: documentation issue?

I have a mix of services with only some of them having a health check. So I configured autoheal like this in my docker-compose.yml:
autoheal:
  image: willfarrell/autoheal:1.1.0
  tty: true
  restart: unless-stopped
  environment:
    - AUTOHEAL_CONTAINER_LABEL=autoheal
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock

And the services with healthcheck got the label:
geoserver:
  image: ...
  labels:
    autoheal: true

Autoheal didn't seem to be working for me. No logs. Well, the instances that could need to be autohealed rarely failed, which is good news, but made it harder to see if autoheal was working.
It turned out that autoheal: true was the problem. true needs to be quoted: autoheal: "true", as autoheal searches for the lowercase value. Just true gets translated to a True boolean by docker compose, which autoheal doesn't search for.
So....

Should autoheal also accept the True?
Perhaps it is enough to just add a quick note to the README, warning about the quotes?
A short message upon startup (echo "autoheal is running") would personally have helped me to be sure the logging was actually working. I never got any output out of the tool because of the true/True mismatch, but I spend quite some time figuring out whether there was actually something wrong with my logging. That's why the tty: true is in there, for instance.

Unrelated to the documentation issue... I could probably have omitted the label entirely in my case? I only used it to restrict autoheal to my containers with a health check, preventing autoheal from looking at the non-health-check containers.
Autoheal searches for {"health": "unhealthy"}, so containers without a health check are completely ignored in any case, it seems.

this certainly can be an annoying, I recall docker-compose has got tendencies to do conversions on yml which IMO it shouldn't.
I'd be nice to turn autoheal check to 1/True/true/yes like.

What is the status of this issue? I am trying to implement this container with haugene/docker-transmission-openvpn, and I think this issue is preventing it from working.
docker-transmission-openvpn has the label built in, but not in quotes so this may be keeping it from getting healed.


Missing a field in Entity Linking datasets

Here is the data example of EL provided in the README:
'23235546-1', # table id
'Ivan Lendl career statistics', # page title
'Singles: 19 finals (8 titles, 11 runner-ups)', # section title
'', # caption
['outcome', 'year', ...], # headers
[[[0, 4], 'Björn Borg'], [[9, 2], 'Wimbledon'], ...], # cells, [index, entity mention (cell text)]
[['Björn Borg', 'Swedish tennis player', []], ['Björn Borg', 'Swedish swimmer', ['Swimmer']], ...], # candidate entities, this the merged set for all cells. [entity name, entity description, entity types]
[0, 12, ...] # labels, this is the index of the gold entity in the candidate entities
[[0, 1, ...], [11, 12, 13, ...], ...] # candidates for each cell

However, the final field:

[[0, 1, ...], [11, 12, 13, ...], ...] # candidates for each cell

is only provided in the test split, while in the train and dev split, it is missing. How to generate this field?

I'm trying to understand the same here...
cc @xiang-deng @huan-sunrise

Hi, as you can see in https://github.com/sunlab-osu/TURL/blob/bfec92e942a648695b3910aab42a6f0b679d37fc/data_loader/EL_data_loaders.py#L28
The field is not used for training. If I recall correctly, when tuning the model, I compute the loss against all candidates for the table, not individual cells, as it is more efficient.
The field is used at test time to compute the final metric, i.e. if the model predicts something that is not in the candidate set associated with the specific cell. We can ignore it. As such we only provide it for the test set. The logic is in evaluate_task.ipynb and data_processing.ipynb.
Let me know if you have other questions.


2.5.1: Pre-compiled version for Visual Studio 2019

Hello all,
I was wondering if anyone knows what the process would be to add a Visual Studio 2019 pre-compiled library on the website (like there is for the other VS versions)
VS2019 came out after 2.5.1, so I'm not sure if we're just waiting on the next stable version to get VS2019 on board, or if it just isn't supported yet.
I have downloaded VS2019, and have successfully compiled it. I wouldn't mind running any tests and/or doing any other required work to make it happen if someone pointed me in the right direction.

Visual Studio 2015, 2017, and 2019 are binary compatible, which is the reason there's no new/extra download. Although this could be marked more clearly on the downloads page.

Ah, this explains it. Thanks!


Perfect Paragon

text: A minion with negative intercept amount cant block an action with zero stealth
link: https://groups.google.com/g/rec.games.trading-cards.jyhad/c/ze7FHSIdaA4/m/5l5rSNheHfsJ

This is not a proper ruling. https://groups.google.com/g/rec.games.trading-cards.jyhad/c/RY27OU0LoTU/m/g3JgxrQfQMsJ would have been (it needs to be a post from a Rules Director / Rulings Ref / Rulings Team)

Stealth, Intercept and Bleed and Hunt amounts can be negative, because nothing in the rulebook says otherwise. In the next rulebook iteration, hopefully, this gets included more clearly. In the meantime, I'd rather wait because adding a ruling for this would take time (there are dozen of cards providing potential negative values), only to remove them once the rulebook is updated.

Now more explicit in the v1.1 rulebook:

Additionally, some cards and other effects can be used to increase or decrease (even below 0) a minion’s stealth or intercept, as noted on card text.


CIVET: 'Test linux' failure

Testing 95cbab1
Test linux : Failed
View the results here.

The 'Test linux' have been fixed and this issue can be closed now.


rbenv version-name > .ruby-version write system to file instead of current version

My current global Ruby version is 2.5.0
rbenv version
=> 2.5.0 (set by /home/xxxx/.rbenv/version)

rbenv version-name
=> 2.5.0

However when I try to redirect that to a file, it outputs system!
rbenv version-name > .ruby-version
cat .ruby-version
=> system

This seems wrong.......
I'm currently on the master branch of rbenv. Commit ID as of now is b943955dbf10c436a8d4c7c36fd4216a34ef2b33.
Thanks

@ayqazi
I created a PR to address the issue you're having.
In the meantime, use the builtin method instead.
rbenv local $(rbenv version-name)
# or simply
rbenv local 2.5.0


TypeError: buffer is too small for requested array

I tried to convert orignal model to ONNX models.
But, the eroor occurs when I executed
!python yolo_to_onnx.py -c 4 --model yolov5-custom
error message
Building ONNX graph...
Traceback (most recent call last)  File "yolo_to_onnx.py", line 1065, in <module>
    main()
  File "yolo_to_onnx.py", line 1053, in main
    verbose=True)
  File "yolo_to_onnx.py", line 585, in build_onnx_graph
    params)
  File "yolo_to_onnx.py", line 459, in load_conv_weights
    conv_params, 'conv', 'weights')
  File "yolo_to_onnx.py", line 487, in _create_param_tensors
    conv_params, param_category, suffix)
  File "yolo_to_onnx.py", line 517, in _load_one_param_type
    buffer=self.weights_file.read(param_size * 4))
TypeError: buffer is too small for requested array

I have verified that cfg and weights are compatible.
Even if I look at the past issues, I don't know how to deal with it, so I asked a question.
I would be grateful if you could let me know if there is any other reason.
Thank you in advance.

TypeError: buffer is too small for requested array
This means the yolo_to_onnx.py code failed to parse and load the cfg &  file.  It's likely due to some parameters (which affects weight sizes) in certain layers which are not implemented in yolo_to_onnx.py.

Original model cfg is below.

I will do some more research.

Yes, it could be due to padding/pad settings in convolutional layers.  Refer to the source code below.  "yolo_to_onnx.py" assumes all convolutional layers are using "SAME_LOWER" padding.  It does not reference the actual padding/pad parameter in the cfg file at all...
https://github.com/jkjung-avt/tensorrt_demos/blob/b757be60af727adf50690fc19209e63d366d8e51/yolo/yolo_to_onnx.py#L683-L723

I replaced SAME_LOWER into NOTSET in code.
But same erroe occured.
So I replaced pad into padding in cfg and train from scratch.
After training, I will tell results as soon as possible.
Thank you in advance.

My model has finished training and I tried to convert onnx model.
But, same error occuerd.
I checked all conv paremeter pad=1 in cfg.
Would you teach me other causes?
Thank you in advance.
'''
Traceback (most recent call last):
File "yolo_to_onnx.py", line 1065, in
main()
File "yolo_to_onnx.py", line 1053, in main
verbose=True)
File "yolo_to_onnx.py", line 585, in build_onnx_graph
params)
File "yolo_to_onnx.py", line 459, in load_conv_weights
conv_params, 'conv', 'weights')
File "yolo_to_onnx.py", line 487, in _create_param_tensors
conv_params, param_category, suffix)
File "yolo_to_onnx.py", line 517, in _load_one_param_type
buffer=self.weights_file.read(param_size * 4))
TypeError: buffer is too small for requested array
'''

I checked all conv paremeter pad=1 in cfg.

The above is not correct.

For convolutional layer with size=1, pad should be 0.
For convolutional layer with size=3, pad should be 1.
For convolutional layer with size=5, pad should be 2.
For convolutional layer with size=7, pad should be 3.
......

In other words, the "SAME_LOWER" padding would produce the same output dimension if stride is 1.

I found pad in darknet cfg.(https://github.com/AlexeyAB/darknet/wiki/CFG-Parameters-in-the-different-layers)

pad=1 - if 1 will be used padding = size/2, if 0 the will be used parameter padding= (0 by default)

My model used size=3 or 1, so setting pad=1 means no probelm.
Also, I found error conv in my cfg.
error
01_convolutional
002_convolutional
003_convolutional
・・・・・・・・・・・
067_convolutional
069_convolutional
TypeError: buffer is too small for requested array

069_convolutional is below
[convolutional]
batch_normalize=1
filters=64
size=1
stride=1
pad=1
activation=swish

This means that other conv parameters is correct, don't you?

Maybe, I think DepthwiseConv cannot correspond to onnx model.
If it is true, please teach me how to convert groupconv block to onnx.
Thank you in advance.

Maybe, I think DepthwiseConv cannot correspond to onnx model.

The "yolo_to_onnx.py" code indeed does not handle DepthwiseConv at all.  Such layers would be converted to normal Conv., which consumes more weights values than DepthwiseConv.  So it's no surprise that you encounter "TypeError: buffer is too small for requested array" when you have DepthwiseConv in the cfg file.

The "yolo_to_onnx.py" code indeed does not handle DepthwiseConv at all. .

I can understand the cause of error.
If you have a time, please teach me how to convert DepthwiseConv to onnx model.
Thank you in advance.