feat: don't force plain themes

is-plain-object is still required as an additional dependency, in order to guarantee that objects within rulesets are toStringed.
Fixes #1776.

@phyllisstein try merging from master - I turned off danger for the moment. Please do add a CHANGELOG entry though 😃

I think we should use this opportunity to get rid of such a small dependency that will likely never change. I think it’d make sense to check for:

not equal null
not an array
is type of object

Without taking on a small dependency. I’m not even sure if we still need this constraint even?

Since this test just passed, I take it SC can pretty well handle non-plain objects:
  it('should not allow other complex objects to be passed as themes', () => {
    class Theme {
      constructor(borderRadius) {
        this.borderRadius = borderRadius
      }
    }

    const theme = new Theme('2px')

    const Comp1 = styled.div`
      border-radius: ${ ({ theme }) => theme.borderRadius };
    `

    const wrapper = mount(
      <ThemeProvider theme={ theme }>
        <Comp1 />
      </ThemeProvider>
    )

    expectCSSMatches(`.sc-a {} .b {border-radius:${theme.borderRadius};}`)
  })

I'd be happy to remove the check---and the dependency---altogether. If you want to keep it, though, isobject isn't the right module.

Consider it done! With a few extra tests to boot.

@phyllisstein great! could you please merge from master to fix CI and add a changelog entry?

With pleasure. Should be ready to go once tests pass.

Looks like there were some Flow issues that I'll work on correcting, but also a problem with the bundle size check that I'm less sure how to resolve.

Heh, Flow is doing it's job and preventing you from making the mistakes! Since these tests are intentional, you can put a // $FlowFixMe above the lines in question to disable Flow for that particular case.

Thanks! With that added and a small bump to the maximum bundle size I think I'm set.

Thank you so much for helping us improve styled-components! Based on our Community Guidelines every person that has a PR of any kind merged is offered an invitation to the styled-components organization—that is you right now!
Should you accept, you'll get write access to the main repository. (and a fancy styled-components logo on your GitHub profile!) You'll be able to label and close issues, merge pull requests, create new branches, etc. We want you to help us out with those things, so don't be scared to do them! Make sure to read our contribution and community guidelines, which explains all of this in more detail. Of course, if you have any questions just let me know!

Released in https://github.com/styled-components/styled-components/releases/tag/v3.3.2!

Neat-o! Thanks for all your help!


Make/cargo build issue

I'm seeing the following when I run make in this project:
$ make
cargo build --target wasm32-wasi --release
error: failed to parse manifest at `/Users/vdice/go/src/github.com/fermyon/bartholomew/Cargo.toml`

Caused by:
  namespaced features with the `dep:` prefix are only allowed on the nightly channel and requires the `-Z namespaced-features` flag on the command-line
make: *** [build] Error 101

Not sure if an update/upgrade to my local rust/cargo env is necessary -- or if an update to this project's Makefile/Cargo.toml is due.

I got it working/building after I ran the following on my machine:
$ rustup install nightly
$ rustup default nightly

If this is the correct solution, should we add something to the docs?

I ran into this today as well. we should document this in README.

i had to also add -f docs/spin.toml to Makefile's  serve target to get the site up and running.

As of writing when building at the HEAD commit of this repo, I no longer appear to need to switch to nightly.  Are you seeing the same @rajatjindal ?  If so, we should be able to close this one out.

I also don't need to use Rust Nightly.
Closing, but feel free to reopen if this is still an issue.
Thanks!


elasticsearch storage example

Adds ElasticSearch as a storage backend along with some basic nip50 search support.

This is awesome.


Rejecting m-lines

Related to #187, I noticed that when Str0m doesn't match any codecs for an m-line it still indicates that the m-line is okay to use.
I believe str0m should be rejecting the m-line by setting the port to 0 in the resulting SDP.
This is specified in section 5.3 of JSEP

For each offered m= section, if any of the following conditions are true, the corresponding m= section in the answer MUST be marked as rejected by setting the port in the m= line to zero, as indicated in [RFC3264], Section 6, and further processing for this m= section can be skipped:

The associated RtpTransceiver has been stopped.
None of the offered media formats are supported and, if applicable, allowed by codec preferences.
The bundle policy is "max-bundle", and this is not the first m= section or in the same bundle group as the first m= section.
The bundle policy is "balanced", and this is not the first m= section for this media type or in the same bundle group as the first m= section for this media type.
This m= section is in a bundle group, and the group's offerer tagged m= section is being rejected due to one of the above reasons. This requires all m= sections in the bundle group to be rejected, as specified in [I-D.ietf-mmusic-sdp-bundle-negotiation], Section 7.3.3.

(emphasis mine)

Sure. Let's reject it. I guess we should both reject on SDP level, and also for Rtc::media()


GEO and Groups

GEO Admins cannot manage O365 Group connected SharePoint sites but can add themselves as a Site Collection Admins to a O365 GROUP.  Change Opening sentence to:  “You can configure separate administrators for each geo location that you have in your tenant. These administrators will have access to the SharePoint Online and OneDrive settings that are specific to their geo location and can add themselves as Site Collection Admin's via PowerShell “Set-SPOUser” cmdlet to O365 Group connected sites.”

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 3563cead-033f-932d-6a8b-e34ab4cc68da
Version Independent ID: 21a1502b-efa3-df19-4176-423c43fe3e4e
Content: Add or remove a geo administrator
Content Source: Enterprise/add-a-sharepoint-geo-admin.md
Service: o365-solutions
GitHub Login: @MikePlumleyMSFT
Microsoft Alias: mikeplum

@JT-Serco-2019  - Thank you for submitting feedback.
I will get this issue over to the Enterprise writing team for investigation.
Thank you for reporting and making the docs better. Much appreciated.
I made a note to request the team to update this when the work is complete.

Hello @JT-Serco-2019 I am proposing to the authors of the article adding this information to the article. Let's see if they approve it.
Thanks for your feedback!

The behavior here is the same as a non-multi-geo environment. SharePoint admins can add themselves as site or group admins in the SharePoint admin center as needed.

Hello @JT-Serco-2019
As @MikePlumleyMSFT explained, this is the usual behavior so no need to add this to this article, however, your contribution will be visible and remain at the bottom of the article.
Many thanks

Serco Business
Mike,
I hate to say it or they misunderstood, a Geo Admin are not assigned as SharePoint Admins because SharePoint Admins have access to all GEO Locations, GEO admins only have access to sites on their GEO and cannot add themselves to a O365 Group  in their GEO, they can add themselves to the SharePoint site as a Site Collection Admin by using PowerShell command only. There is no GUI interface.
JT
James K. Turner, PMP & Security+
Systems Administrator IV &
Collaboration Tech Lead,
Microsoft 365 SharePoint
Enterprise Applications Serco Inc.
T:<PHONE_NUMBER>
james.turner<EMAIL_ADDRESS>www.serco-na.comhttp://www.serco-na.com/
<EMAIL_ADDRESS>From: Andres Mariano Gorzelany<EMAIL_ADDRESS>Sent: Friday, October 18, 2019 6:13 AM
To: MicrosoftDocs/OfficeDocs-Enterprise<EMAIL_ADDRESS>Cc: Turner, James, K (Serco NA<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: [EXTERNAL] Re: [MicrosoftDocs/OfficeDocs-Enterprise] GEO and Groups (#559)
Attention: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hello @JT-Serco-2019https://github.com/JT-Serco-2019
As @MikePlumleyMSFThttps://github.com/MikePlumleyMSFT explained, this is the usual behavior so no need to add this to this article, however, your contribution will be visible and remain at the bottom of the article.
Many thanks

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/OfficeDocs-Enterprise/issues/559?email_source=notifications&email_token=ANOW26626RFCZV4IG3BK2WDQPG77BA5CNFSM4I76YPR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBUTMRY#issuecomment-543766087, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANOW266NGPNKHWK5DCASPH3QPG77BANCNFSM4I76YPRQ.

@MikePlumleyMSFT - Can you please share your inputs on this additional feedback?
Thank you.
cc: @kenwith

@JT-Serco-2019 - We appreciate your efforts to clarify the docs.
Regarding your additional feedback, we have reached out to the Product team and we were told that the usual admin activities are available in the UI for geo-administrators for sites in their geo location. In any case, the PowerShell option is really outside the scope of the document.
Thank you for your contribution to make the docs better! Much appreciated!


Normalise info to standard strings

There are currently many different ways to refer to roughly the same things, from rip types to language/subtitle names.
There should be a mapping to a consistent set of values from the different regexes currently available: 'ENG-CHS' could become ["English", "Chinese"] - all the WEB-DL, WEB-DLMUX, WEB-DLMXX could refer to 1 thing (whatever that is). This would likely become the default behaviour, and an optional parameter to parse() could be added to revert back to just returning the literal string in the torrent name.
This could be done by changing the patterns.py file to use a dict, with the same keys, and the values would become tuples of the current patterns together with their standardized representation.

Fields that would need this (on top of general capitalization fixes):

audio e.g. "LINE" to "LiNE", possibly harder cases.
codec e.g. "x264" to "H.264".
language e.g. "eng" to "English", ["ita", "eng"] to ["Italian", "English"].
subtitles same as above.
quality e.g. "WEB" and "WEB-DL" both to "WEB-DL".
resolution e.g. "1280x720" to "720p".
size maybe change to an integer, always return in bytes? Otherwise leave the same.


Can resource data in binary file format be loaded from FileX?

I have external NOR Flash that is formatted with Azure FileX, and am wondering if I can use gx_binres_theme_load or similar to load a resource data binary file that is stored on the filesystem?
From my understanding I would need to expose the file's data as a raw pointer, which would require copying the file to RAM first (not an option given RAM constraints). Is there some functionality I'm missing that could make this work?

gx_binres_theme_load isn't designed to work from a file. The function dynamically creates resource tables, i.e. tables of pointers to the resource data. Pointers of course must point to directly accessible memory, not to a filesystem. People usually use flashing tools provided by their silicon provider or with their debugging tools to program the resource data to the external flash memory. Can you reserve a block of your flash for this purpose, rather than allocating the entire memory to FileX?

The external flash we have is controlled over the FSMC bus, and involves setup commands to place into read-mode before reading a section of data. Heres the snippet from the STM32F4 HAL library HAL_NOR_ReadBuffer func:
  /* Send read data command */
  NOR_WRITE(NOR_ADDR_SHIFT(deviceaddress, uwNORMemoryDataWidth, NOR_CMD_ADDRESS_FIRST), NOR_CMD_DATA_FIRST);
  NOR_WRITE(NOR_ADDR_SHIFT(deviceaddress, uwNORMemoryDataWidth, NOR_CMD_ADDRESS_SECOND), NOR_CMD_DATA_SECOND);
  NOR_WRITE(NOR_ADDR_SHIFT(deviceaddress, uwNORMemoryDataWidth, NOR_CMD_ADDRESS_THIRD), NOR_CMD_DATA_READ_RESET);

  /* Read buffer */
  while( uwBufferSize > 0U)
  {
    *pData++ = *(__IO uint16_t *)uwAddress;
    uwAddress += 2U;
    uwBufferSize--;
  }

Perhaps gx_binres_theme_load would work if we configured the flash for 8bit reads and left it in read-mode, however FileX would be regularly performing reads/writes to the same peripheral, and I doubt they will play nicely together.
We may need to make some hardware changes for this to work. Is there an example project which utilises loading resources from external flash that I could refer to?

If you clone or have cloned the GUIX repository (GUIX Studio will do this for you), you should see guix/tutorials/demo_guix_binres, which is configured to run either on the desktop (using a filesystem to load the resource data to a block of RAM) or the STM32_F439xx target board (resource data pre-loaded to NOR flash). This should be a useful example for you.
We are working with another customer on supporting a "data read callback" function to allow reading resource data from external serial flash, however the initial design is to support only a few custom canvas drawing APIs (gx_canvas_streamed_pixelmap_draw(), etc..)  and not full widget set support for this model. I don't think this model is feasible from a performance perspective, especially as relates to glyph drawing, without some data caching mechanism. For the near future we really need to be able to directly access the resource data without the overhead of a "data read callback".

Thats understandable, I appreciate the help! The demo_guix_binres STM32F439 project uses a similar NOR Flash configuration as my project which is very convenient. Like you originally said if I can have a section of the NOR Flash for binres and the remainder for FileX then that will be the ideal solution; I'll just need to be confident that they dont clash when FileX needs to performs writes etc.
Thanks again for the help :)


qemu 5.0.0 2 build error

@jedavies-dev
ERROR: sizeof(size_t) doesn't match GLIB_SIZEOF_SIZE_T.
You probably need to set PKG_CONFIG_LIBDIR
to point to the right pkg-config files for your
build target
qemu-2020-06-27-13:14-2408.txt
had 5.0.0 1 installed with no problems, issue confirmed by micr0

Interesting - just tried rebuilding this and it works OK for me on my normal KISS installation.  Also tried this from a clean chroot (starting from the release tarball, add community repo, do "kiss b qemu").  Did this on powerpc64le and x86_64.  Package built OK in each case.
Have you updated glib recently?  There was a change to glib to enable building of static libraries that qemu now depends on.  Qemu would fail if glib isn't on the latest version for this reason.

i was doing a kiss update when it failed to build so glib is up to date

ok after rebuilding/installing glib it now works, closing


fix: lodash.pick vulnerability

Context
Solving high security issues:
https://github.com/ImagingDataCommons/Viewers/security/dependabot/198[High](https://github.com/ImagingDataCommons/Viewers/security/dependabot?q=is%3Aopen+severity%3Ahigh)
#198 opened 2 weeks ago • Detected in lodash.pick (npm) • yarn.lock
https://github.com/ImagingDataCommons/Viewers/security/dependabot/197[High](https://github.com/ImagingDataCommons/Viewers/security/dependabot?q=is%3Aopen+severity%3Ahigh)
#197 opened 2 weeks ago • Detected in lodash.pick (npm) • docs/latest/yarn.lock
https://github.com/ImagingDataCommons/Viewers/security/dependabot/196[High](https://github.com/ImagingDataCommons/Viewers/security/dependabot?q=is%3Aopen+severity%3Ahigh)
#196 opened 2 weeks ago • Detected in lodash.set (npm) • yarn.lock
Refer to https://github.com/lodash/lodash/issues/5809#issuecomment-1910560681 to understand the reason for the new resolution to be a url and not a number.
Changes & Results

Testing

Checklist
PR

[] My Pull Request title is descriptive, accurate and follows the
semantic-release format and guidelines.

Code

[] My code has been well-documented (function documentation, inline comments,
etc.)

Public Documentation Updates

[] The documentation page has been updated as necessary for any public API
additions or removals.

Tested Environment

[] "OS:
[] "Node version:
[] "Browser:

Please try using a published version that contains the fix instead of archived versions.

@igoroctaviano if you refer to the link I posted in the description of the PR you'll understand why it's not possible as of now to do what you ask. Regards.

Can we use the second approach from that thread?
As @nclavaud notes, the modularised packages are now essentially deprecated. Thus, If you use lodash.pick or another modularised lodash subset as a direct dependency in your projects, drop it and replace it with lodash. If you're able to apply it to transitive uses, the solution suggested by @Pfahlf makes sense.

Maybe we just need to bump to the latest version of lodash and use lodash instead of modularized subset import.

Can we use the second approach from that thread?
As @nclavaud notes, the modularised packages are now essentially deprecated. Thus, If you use lodash.pick or another modularised lodash subset as a direct dependency in your projects, drop it and replace it with lodash. If you're able to apply it to transitive uses, the solution suggested by @Pfahlf makes sense.

Maybe we just need to bump to the latest version of lodash and use lodash instead of modularized subset import.

@igoroctaviano I wouldn't know how to do that since we are not directly using lodash.pick, instead, the presence of lodash.pick comes from a long chain of dependencies from other packages:
@ohif -> i18n -> locize-cli -> android-string-resource -> rdotjson -> cheerio -> lodash.pick
So I don't know how I'd force a package to use another lib instead of the one that's specified in that package's package.json, unless we do it the way I have done it, which is essentially doing that by telling it to download lodash from the url I provided, instead of downloading any version of lodash.pick.
FYI I also tried updating locize-cli to the latest version to see if it would help but it doesn't make a difference in this case.


When to support map type?

Just realized that map type is not supported by the current version. If you select from table it would show "Unknow data type map". Best regards.

No ETA yet, PR is welcome.


[Task]: Fix client: no password has been supplied, or the key has been cleared from the memory

Task description
Sometimes the password is cleared from the memory blocking the entry to profiles.
The following error is thrown, after which the given Firefly profile is locked:
client: no password has been supplied, or the key has been cleared from the memory
Users have already experienced it:
https://discord.com/channels/397872799483428865/748265907351978115/1069927640103915571
Requirements
TBD
Creation checklist

[ ] I have assigned this task to the correct people
[X] I have added the most appropriate labels
[X] I have linked the correct milestone and/or project

Might have to do something with wallet.rs updates in between apps. Test by downloading alpha-2, creating a new profile and then updating to alpha-3.

When we try to login after restoring a profile from a mnemonic with no connection to the node. The API call that fails it the createAccount one.
The error is thrown from iota.rs from the following functions:

StrongholdAdapter get() function
StrongholdAdapter insert() function
read_stronghold_snapshot() which is also called by change_password()
write_stronghold_snapshot()
generate_addresses()
signature_unlock()
[store_mnemonic()](https://github.com/iotaledger/iota.rs/blob/f2ff122e110f3645f30a449722ed9a6470f731d3/client/src/stronghold/secret.rs#L209


Some viz

I've made some visualizations of the data here that may help folks get their heads around what's here.
See https://estaub.github.io/covid-project-report/ .
Feel free to steal, but let me know if you decide to publish similar work, so I'll know to leave a tombstone with a link to yours.

cc: @gregce

cc: @gregce

WELCOME TO SHOPIFY,KOSSI
https://en-dieu-nous-croyons.myshopify.com CUSTOMIZE YOUR ONLINE STORE TO MATCH YOUR BRAND .
E-COMMERCE INSPIRATION
05/05/2018 13H37 WITH WORLDBANK
ADD OR BUY A CUSTOM DOMAIN
www.amazon.tg


[GLUTEN-1632][CH]Daily Update Clickhouse Version (20241024)

Auto commit by gluten daily build, please check the build status and merge it if it's green.

Run Gluten Clickhouse CI


Add float literal to the new parser

Description:
Refer: https://ballerina.io/spec/lang/2020R1/#floating-point-literal

Duplicate of https://github.com/ballerina-platform/ballerina-lang/issues/22358


No need for supporting a fork

It looks like it is possible to do everything Spreads-specific, inclusing #1, without accessing internals of R.NET. Should try to avoid any modification, use git rebase or just use a binary dependency.

Upstream is dead, long live R.NET!


drivers: gpio: Make PCA9535 driver generic to support multiple PCA95xx devices

PCA9535 and PCA9555 IO-expanders share most of functionality and registers, the intention of this change is to make existing driver generic for both devices.  The alternative will be to add a new PCA9555 driver with mostly same functionality that existing one.
PCA9535 delta features are: programmable output drive strength, latchable inputs,
pull-up/pull-down resistors, maskable interrupts and drain or push-pull outputs. From these current gpio driver only uses the support for pull-up/pull-down. The new KConfig GPIO_PCAL95XX_PUD allows to disable this feature when PC9555A is used.
Note: Original driver is not tracked in code owners hence adding owners based on git log history.
Signed-off-by: Jose Alberto Meza<EMAIL_ADDRESS>

@albertofloyd we need to move this driver to the new GPIO API, see https://github.com/zephyrproject-rtos/zephyr/issues/18530
same for the XEC gpio driver (gpio_mchp_xec.c)

@nashif  can this additional KConfig been included in the porting process for this driver?
@franciscomunoz  can you take a look into changes needed for gpio_xec?


'Disable / Enable multichannel metering' actions: improve performance

report / discussion here:
https://forum.cockos.com/showthread.php?p=2090523#post2090523
+ small Auto group tweaks

I think the performance hit is not the parsing itself, it's passing a new chunk back to reaper - it doesn't know what exactly you've changed so it has to update the entire track state.  In a chunk with plugins that can get expensive fast.

Can you update the whatsnew.txt please?

You're welcome.  Now that there's not a huge intimidating pile of PRs to look at I should be able to address these faster.


[sram] Initialize the AST from the flash info page

Check if there is AST calibration data in info page 0, and if so, write it to the AST registers.

CC: @benjbender this should help when you load SRAM programs, it will automatically config AST for you if you have already setup the flash info page 0 correctly


Document that we build/deploy using ARM

Originally posted by @GaryGSC in https://github.com/byu-oit/hw-fargate-api/issues/686#issuecomment-1291178681

I want to link back to issue #389, but any links I drop in the README wouldn't survive the find-and-replace step. I guess that's fine because some other links (like https://github.com/byu-oit/hw-fargate-api/generate) also break at the same time.

Closed via https://github.com/byu-oit/hw-fargate-api/pull/693/commits/1f4b4b61eeb1a02d770cf1aa99c79c4db9006ad9 as part of #693.


About student time scale

Very nice work!
A little question is why the time of studnet is "t - self.time_scale", i think it may be "t"?
Link: https://github.com/Rongjiehuang/ProDiff/blob/1d199bc504afed8756d62bf108c460b1e89330f8/modules/ProDiff/model/ProDiff.py#L172

Because we need a teacher to distill the knowledge to a student, which have different time scales (T=2 vs. T=4)


sizeof: new package with constants for Int, Uint, Uintptr, Int8, Float64, etc

See the tail of the (closed) proposal #5602. There I suggested that instead of making unsafe.Sizeof a safe thing, which it still kinda isn't in general, we solve 90% of the problem by add easy-to-use constants to a safe package, such as:
package reflect

const (
  SizeofInt = <size on this machine>
  SizeofUint = SizeofInt
  SizeofBool = 1
  SizeofFloat64 = 8
  //...
)

These would be defined for primitive types only, not slices, maps, etc. You'd still need the unsafe package to handle them.

@robpike have your feelings on this issue changed at all?

I still think it's wrong that the library says that to find the size of int is unsafe: calling unsafe.Sizeof(int) just isn't unsafe. You can compute its size, but it's really tricky and hard to work out from first principles (see the constants blog post for the mechanism).
I still think a handful of constants in someplace safe is a friendlier solution than using unsafe.Sizeof for this subset of its use.

we solve 90% of the problem by add easy-to-use constants to a safe package, such as:
package reflect

const (
  SizeofInt = <size on this machine>
  SizeofUint = SizeofInt
  SizeofBool = 1
  SizeofFloat64 = 8
  //...
)

These would be defined for primitive types only, not slices, maps, etc.

Nearly all the primitive types are already sized (uint8, float64 etc). The only unsized types are int, uint, uintptr, and bool, right? I am skeptical that there is so much code doing unsafe.Sizeof(true) to find out how big a bool is. Checking the size of an int is certainly common. bits.UintSize is the size of an int or uint in bits (but its bits).
Maybe we could add to reflect just:
const (
    IntSize = ...
    PtrSize = ...
)

Maybe IntSize belongs in math?
Math is mostly about floats, but it does have MaxInt8 etc.
Maybe PtrSize belongs in runtime?
It would be unfortunate for code to import reflect just to find out how big an int is (not as unfortunate as importing unsafe but still unfortunate).

@robpike what do you think?

Sounds fine. I'd be fine with them all in reflect - it's a reflective operation to ask a question about a type, and reflect is part of everything that formats values so it's not a burden to the binary - but am open to other options. Maybe even a standalone tiny package of constants.

They could even go into builtin...

I'm not sure if the ship has sailed on the package to which these belong, but doesn't it seem inconsistent to already have the bit size for a builtin in math/bits (uint), and still declare the bit sizes for int and uintptr in separate packages? I see math/bits as the singular location containing operations and constants to deal with integral types at the bit-level. Declaring these in separate packages would only serve to confuse users and add to the number of dependencies.

Summary of discussion so far:
I think from the original proposal we're down to IntSize, UintSize, and PtrSize.
For location, math/bits doesn't seem perfect but bits.UintSize already exists.
We could put them elsewhere but no other package seems better.
We don't want to import reflect just for sizes.
Using runtime (next to runtime.GOARCH) would be defensible but maybe not better than math/bits.
Math would be a bit odd since there is nothing about unsized integers in its API today.
And 'builtin' is not a real package at the moment.
Maybe math/bits is the best place.
bits.UintSize already exists.
Do we need to add bits.IntSize too?
bits has nothing to do with pointers, so bits.PtrSize would be a mistake.
Probably reflect.PtrSize is the answer there? Reflect is all about pointers.
So it looks like the open question are:

Do we add bits.IntSize = UintSize ?
Do we add reflect.PtrSize = ... ?

If you put UintSize into bits, I'd put PtrSize there too so they're in one place.

@robpike, sorry for the delayed response. I am worried about the message putting PtrSize into math/bits sends. Math/bits is entirely arithmetic operations on integers, hardly any of which are appropriate or even safe to do on Go pointers (maybe TrailingZeros). It would stick out in the API docs ("what is this doing here?"). In contrast, reflect has lots of pointer stuff.

@rsc, I think a comment can cover that well enough, and it sends a nice message to put all the Size constants in one place.

Just to clarify, is the current consensus to change the constant declarations in math/bits to something like the following?
const uintSize = 32 << (^uint(0) >> 32 & 1) // 32 or 64

const ptrSize = 32 << (^uintptr(0) >> 63)

const (
    // UintSize is the size of a uint in bits.
    UintSize = uintSize

    // IntSize is the size of an int in bits.
    IntSize = UintSize

    // PtrSize is the size of a uintptr in bits.
    PtrSize = ptrSize
)

@smasher164, I think the consensus is to add IntSize = UintSize to math/bits.
I still have serious reservations about adding PtrSize anywhere but reflect.
There is not a mention of a pointer anywhere in math/bits and I still believe it's important not to even give a hint that bit tricks on pointer representations is a good idea, which the name bits.PtrSize would.
Looking through the conversation history I am not sure why we started talking about the number of bits in a pointer at all.
@rogpeppe, you suggested adding UintptrSize to math/bits.
What was the motivation behind that suggestion?
When would you use it?
Ian points out that UintptrSize == UintSize now on all supported architectures but of course we might not want to assume that for all time. But still, why care about UintptrSize at all?

Ping @rogpeppe or anyone else for why we need UintptrSize at all.

@rsc For consistency. Why should one numeric type with implementation-defined size be different from the others?

I will reiterate a point I made above: Much of the argument here is a consequence of the decision to put this in math/bits, where I don't think it belongs. Why not create a new package, called say size, and put them there? This also opens a place to put helpful functions to safely compute things like the size of an array or struct or slice or possibly? map.

I have used the size of uintptr in a set type, effectively map[*T]struct{} but with a Reset method to quickly reuse the same memory. Keys were uintptr, coming from e.g. reflect.ValueOf(x).Pointer(). I used a scrambler of the form A*key+C to try to spread out which bits were used for nearby keys; the values of A and C depend on the size of uintptr. I found after pushing that my original implementation didn't compile specifically on 32-bit targets, because the A and C constants overloaded uintptr there; the final implementation uses apparently nonsense type conversions between uint64 and uintptr on a branch that is dead in the cases that require them to be there.
Later, after much more reading, I got the feeling that garbage collection could move objects while I was using such a set, so I made the keys be just a new field on T; the only requirement for them to remain uintptr was that I had already published the set type with that API.
The overall experience suggests to me that anyone who wants to use the size of uintptr probably should know how to find it (although I'd probably say the same is true of bits.UintSize in the first place, so that isn't a real argument). Also, at least in my case, it was easier to do if ^uintptr(0) != 0xffffffff than to care about the exact size.

Talked to @robpike about this. Given the trouble with finding an existing home for these, he suggests a new package:
// Sizeof defines the sizes of basic Go types in bytes.
package sizeof

const (
    Bool = 1
    Int8 = 1
    Uint8 = 1
    ...
    Int = ...
    Uint = ...
    Uintptr = ...
)

These could almost be defined as, for example, Int = unsafe.Sizeof(int(0)), but that would result in Int being a uintptr, not an untyped constant.
Having a separate package eliminates all the problems with none of the existing packages being right.
Retitling for this new idea.

No change in consensus, so accepted.

Considering the help wanted flag and the fact that the discussion has stalled. I thought I would offer an implementation for review: (I took the liberty of adding Interface, Map, Ptr, and Slice, since they are constant size pointer like types, but left off Array and Struct since they are variable size value types.)
package sizeof

const(
        Bool = 1
        Byte = Uint8
        Complex128 = 16
        Complex64 = 8
        Float32 = 4
        Float64 = 8
        Int = word
        Int16 = 2
        Int32 = 4
        Int64 = 8
        Int8 = 1
        Interface = 16
        Map = 8
        Ptr = 8
        Rune = Int32
        Slice = 24
        String = 16
        Uint = word
        Uint16 = 2
        Uint32 = 4
        Uint64 = 8
        Uint8 = 1
        Uintptr = word
        word = 32 << (^uint(0) >> 32 & 1) // 32 or 64
)

We would also want to fixup math/bits to read:
package bits

// UintSize is the width of a uint in the current runtime.
//
// Deprecated: please use sizeof.Uint.
const UintSize = sizeof.Uint

Interface, Slice and String have different sizes on 32 bit architectures.

Good catch, what should they be? My guess is that we could record them in relation to word:
Interface = word / 4
Slice = word * 2 / 3
String = word / 2

Was it considered to add a new builtin function sizeof that can accept both type and expression arguments (like C's sizeof operator) and returns untyped constants (so it's usable where unsafe.Sizeof isn't)? E.g., sizeof(int) seems as friendly as sizeof.Int, and it generalizes easily to non-primitive types (e.g., #40169).

CL 242783 is a proof-of-concept for adding a new sizeof builtin variant of unsafe.Sizeof that returns untyped constants, and extends unsafe.Alignof and unsafe.Sizeof to accept type expressions as arguments in addition to value expressions.

Notably, I proposed extending unsafe.Alignof and unsafe.Sizeof to accept type arguments back in 2015: https://github.com/golang/go/issues/12994
It was rejected because "I think this simplifies very few real programs." Where's the data that sizeof.Int simplifies more than a few real programs?
The regular expression I posted in #12994 now lists 166 instances of unsafe.Alignof/unsafe.Sizeof within the standard library (up from 105 in 2015). It finds 83 instances in third-party Go libraries used internally within Google (file:third_party/golang).
By comparison, CL 242018 finds just 6 instances in the standard library that benefit from this. The same internal Google codebase has just 12 instances of ^uint(0) >> 63 (uint\(0\)\ *>>\ *63). Of those 12, only 1 actually needs to be untyped, and only 1 would require an explicit conversion from uintptr to int to pass to strconv.ParseInt if replaced with unsafe.Sizeof. The rest are all either explicitly converted anyway, or only used in the RHS of a shift operation. (1 was completely unused, and another appeared in a comment.)

@mdempsky, commenting on an accepted issue to suggest new ideas isn't something we typically see.
It would be better to file a new proposal suggesting exactly what you want to propose.
I don't see a description of what you're proposing, only a CL.
After you file it, please move it into the Proposal project Incoming.
Thanks.

FWIW, I worry about sizeof(x) overfitting to expectations from C.
I spoke to Rob about this, since he suggested the package approach in the first place, and he pointed out that having a package means we can extend it in interesting ways in the future, including things like calculating the actual memory size of a linked data structure, or a map, or a slice, or some other thing entirely, none of which is possible if we instead put C's sizeof(x) into the Go spec.

Given the other suggestions that have come in related to this package, it seems like we should move it back into active discussion. In the previous comment I mentioned that having a package means we can have more interesting functionality like sizeof.Map - how much memory does the entire map take up? - and so on. So maybe we should put off introducing these basic constants until we have a full picture of what we want the package to be.
I've put it on hold, though, since we don't have bandwidth to elaborate the other features right now.

By adding SizeofInt, doesn't it mean the arbitrary precision int proposal is given up?

@go101, I don't think that proposal is really viable for the type literally named int anyway.
(It doesn't fit the model of http://golang.org/design/28221-go2-transitions#language-redefinitions.)
I think if that proposal were adopted, the type would have to introduce a new type with a name other than int, in which case sizeof.Int would not be misleading anyway.

How about introduce a new generic builtin function sizeOf:
func sizeOf[T any]() uintptr

How about introduce a new generic builtin function sizeOf:
func sizeOf[T any]() uintptr

Related to https://github.com/golang/go/issues/50019 I guess. The main reason a function is not sufficient is that the result of a function call isn't a constant, so couldn't be used to parameterize array lengths etc.

Related to #50019 I guess. The main reason a function is not sufficient is that the result of a function call isn't a constant, so couldn't be used to parameterize array lengths etc.

Not sure if it can be used as an opportunity to introduce const function


Feature request - Auto restart/refresh feed / note dropped feeds

Sometimes/frequently, my camera feeds timeout (stop receiving data).
Request: Some kind of output noting that the feed has dropped, also an auto-restart feature? (Clicking the top right side MSE/RTC reloads the feed manually.)

I just saw this https://github.com/AlexxIT/WebRTC/pull/622, it's probably what I'm asking for.

Depends on what mode you have the stream in and the reason it stopped. It can be:

broken stream (missing network packets from camera)
camera offline/disconnect
mse player doesn't have time to load data


FileNotFoundException after copying and renaming a cshtml file in a Blazor app

Here's what I did that resulted in this exception:

Create a new Blazor 0.9.0 standalone app in VS2019 Preview channel

Copy and paste Index.cshtml (generated name is Index - Copy.cshtml) and rename it to Todos.cshtml

Update the contents of Todos.cshtml with:
@page "/todos"

Todos

Run the app and browse to /todos

Exception from ActivityLog.xml
Error encountered from project 'C:\Users\daroth\Source\Repos\WebApplication6\WebApplication6\WebApplication6.csproj': System.IO.FileNotFoundException: Could not find file 'C:\Users\daroth\Source\Repos\WebApplication6\WebApplication6\Pages\Index - Copy.cshtml'. File name: 'C:\Users\daroth\Source\Repos\WebApplication6\WebApplication6\Pages\Index - Copy.cshtml'    at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)    at System.IO.FileInfo.get_Length()    at Roslyn.Utilities.FileUtilities.GetFileLength(String fullPath)    at Microsoft.CodeAnalysis.FileTextLoader.ValidateFileLength(Workspace workspace, String path)    at Microsoft.CodeAnalysis.FileTextLoader.<LoadTextAndVersionAsync>d__8.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at Microsoft.CodeAnalysis.Razor.ProjectSystem.DefaultProjectSnapshotManager.<>c__DisplayClass20_0.<<DocumentClosed>b__0>d.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at Microsoft.CodeAnalysis.Razor.ProjectSystem.DocumentState.<GetTextVersionAsync>d__25.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)    at Microsoft.CodeAnalysis.Razor.ProjectSystem.DocumentState.ComputedStateTracker.<GetGeneratedOutputAndVersionCoreAsync>d__7.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at Microsoft.CodeAnalysis.Razor.ProjectSystem.DefaultDocumentSnapshot.<GetGeneratedOutputAsync>d__20.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at Microsoft.CodeAnalysis.Razor.BackgroundDocumentGenerator.<ProcessDocument>d__39.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at Microsoft.CodeAnalysis.Razor.BackgroundDocumentGenerator.<TimerTick>d__43.MoveNext()

@ajaybhargavb could you please look into this? Thanks!

@danroth27, are you able to reliably repro this? I tried your steps and I was only able to reproduce this once in 16.0 Preview and I can't reproduce this at all in 16.1.

@ajaybhargavb From experience I've learned that copying files in the solution is problematic, but no I can't repro this particular issue reliably.

I can't reproduce this in 16.1 even once makes me think we did something to fix this but I'm not sure what. I'll keep trying.

if we can't repro this I don't think it's worth spending lots of time on it unless it comes through VS feedback.
Doing _imports.razor would be more valuable

Agree. Closing this. If there is such an issue, it'll show up again in the future.


Allow to set default gateway

In CI, the default gateway isn't the one associated with the private
network. Allowing to pass it will ensure we don't lose public access.

recheck

Issue is unrelated to the change, re-running "in case I'm lucky" with nested crc gods:
https://review.rdoproject.org/zuul/build/5d33507a78534750957d8c02451cfe82/log/ci-framework-data/logs/ci_make_006_run_edpm_deploy.log#14
@stuggi @fao89 FYI, this is needed for the extracted crc layout, as explained in the commit message. Of course, once we can edit the ansibleVars directly, this kind of change won't be necessary anymore :).
Care to give it a push even if CI fails? It's working on the test patch here:
https://github.com/openstack-k8s-operators/ci-framework/pull/361
We're currently facing the "next error", which is DNS related. We have to kustomize dnsmasq, but that's on us only :).
Thanks!

The jobs all passed independently across two attempts, so we are merging this.


inverting the mcpwm dead-time output is not selective per generator (IDFGH-10050)

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.
fatal: No names found, cannot describe anything. (I'm on latest master).
Operating System used.
Linux
How did you build your project?
VS Code IDE
If you are using Windows, please specify command line type.
None
Development Kit.
esp32 rev1 module
Power Supply used.
External 3.3V
What is the expected behavior?
from the page MCPWM Dead-Times, I feel it is correct to assume that it is possible to selectively invert the output for only one signal but not the other, by setting dead_time_config.flags.invert_output = true; as in the following excerpt from that page:
mcpwm_dead_time_config_t dead_time_config = {
        .posedge_delay_ticks = 50,
        .negedge_delay_ticks = 0
    };
    ESP_ERROR_CHECK(mcpwm_generator_set_dead_time(gena, gena, &dead_time_config));
    dead_time_config.posedge_delay_ticks = 0;
    dead_time_config.negedge_delay_ticks = 100;
    dead_time_config.flags.invert_output = true;
    ESP_ERROR_CHECK(mcpwm_generator_set_dead_time(gena, genb, &dead_time_config));

When I try it myself, I expect it to selectively invert only one signal but not the other.
What is the actual behavior?
Either both signals are inverted simultaneously, or neither.
Steps to reproduce.
This bug exists since a long time, at least many months. Only now have I been able to distill a minimal example that exposes it (it is a c++ project, in case that is relevant):
#include "driver/mcpwm_prelude.h"
#include "driver/gpio.h"

uint32_t SIDES=2;

gpio_num_t pin_L=GPIO_NUM_23;
gpio_num_t pin_H=GPIO_NUM_18;

//uint32_t dead_ticks=0;
uint32_t mcpwm_tmr_rsltn=10000000; // 1 tick = 0.1us
uint32_t mcpwm_period=400; // 25KHz

mcpwm_timer_handle_t mcpwm_timer={};
mcpwm_oper_handle_t mcpwm_operator={};
mcpwm_gen_handle_t mcpwm_generators[SIDES]={};
mcpwm_cmpr_handle_t mcpwm_comparator={};
mcpwm_cap_timer_handle_t mcpwm_capture_timer={};

extern "C" void app_main(void)
{
     mcpwm_timer_config_t timer_config=
    {
        .group_id=0,
        .clk_src=MCPWM_TIMER_CLK_SRC_DEFAULT,
        .resolution_hz=mcpwm_tmr_rsltn,
        .count_mode=MCPWM_TIMER_COUNT_MODE_UP,
        .period_ticks=mcpwm_period,
    };
    ESP_ERROR_CHECK( mcpwm_new_timer(&timer_config, &mcpwm_timer) );

    mcpwm_operator_config_t operator_config=
    {
        .group_id=0,
        .flags=
        {
            .update_gen_action_on_tez=true,
            .update_dead_time_on_tez=true,
        },
    };
    ESP_ERROR_CHECK( mcpwm_new_operator(&operator_config, &mcpwm_operator) );
    ESP_ERROR_CHECK( mcpwm_operator_connect_timer(mcpwm_operator, mcpwm_timer) );

    mcpwm_comparator_config_t compare_config=
    {
        .flags=
        {
            .update_cmp_on_tez=true,
        },
    };
    ESP_ERROR_CHECK( mcpwm_new_comparator(mcpwm_operator, &compare_config, &mcpwm_comparator) );

    int gen_gpios[2]={pin_L,pin_H};
    mcpwm_generator_config_t gen_config={};
    for (int i=0;i<2;i++)
    {
        gen_config.gen_gpio_num = gen_gpios[i];
        ESP_ERROR_CHECK( mcpwm_new_generator(mcpwm_operator, &gen_config, &mcpwm_generators[i]) );
    }

    for (int i=0;i<2;i++)
    {
        ESP_ERROR_CHECK( mcpwm_generator_set_action_on_timer_event(mcpwm_generators[i],MCPWM_GEN_TIMER_EVENT_ACTION(MCPWM_TIMER_DIRECTION_UP, MCPWM_TIMER_EVENT_EMPTY, MCPWM_GEN_ACTION_HIGH)) );
        ESP_ERROR_CHECK( mcpwm_generator_set_action_on_compare_event(mcpwm_generators[i],MCPWM_GEN_COMPARE_EVENT_ACTION(MCPWM_TIMER_DIRECTION_UP, mcpwm_comparator, MCPWM_GEN_ACTION_LOW)) );
    }

    mcpwm_dead_time_config_t deadtime_config_L={.posedge_delay_ticks=1}; // necessary, otherwise deadtimes won't kick in
    deadtime_config_L.flags.invert_output=false;
    ESP_ERROR_CHECK( mcpwm_generator_set_dead_time(mcpwm_generators[0], mcpwm_generators[0], &deadtime_config_L) );
    mcpwm_dead_time_config_t deadtime_config_H={.posedge_delay_ticks=1};
    deadtime_config_H.flags.invert_output=true;
    ESP_ERROR_CHECK( mcpwm_generator_set_dead_time(mcpwm_generators[1], mcpwm_generators[1], &deadtime_config_H) );

    ESP_ERROR_CHECK( mcpwm_timer_enable(mcpwm_timer) );
    ESP_ERROR_CHECK( mcpwm_timer_start_stop(mcpwm_timer,MCPWM_TIMER_START_NO_STOP) );

    mcpwm_comparator_set_compare_value(mcpwm_comparator,mcpwm_period*0.25);
}

Debug Logs.
No response
More Information.
I know you might object that I've got the generators wrong in the call to mcpwm_generator_set_dead_time(...), but not only is the documentation confusing as to what it actually means, and not only does everything else work fine when I use the combination as above, but also there are 16 possible combinations in total, and I've iterated over all of them. Neither one produces the desired output.
So if selective inversion truly is possible, and there is a combination of generators that works, then there must be a bug in the library.

Hi @moefear85 the inverting is not applied to either generator, it's applied to the "deadtime path".

mcpwm_dead_time_config_t dead_time_config = {
        .posedge_delay_ticks = 50,
        .negedge_delay_ticks = 0
};
ESP_ERROR_CHECK(mcpwm_generator_set_dead_time(gena, gena, &dead_time_config));

This way, we're applying the dead time to the gena itself, we're going the gena->gena path, so we're configuring the switch S2 here.
    dead_time_config.posedge_delay_ticks = 0;
    dead_time_config.negedge_delay_ticks = 100;
    dead_time_config.flags.invert_output = true;
    ESP_ERROR_CHECK(mcpwm_generator_set_dead_time(gena, genb, &dead_time_config));

Now we're producing genb from the original gena, so we're going the gena->genb path, so we're configuring the switch S3.
S2 and S3 are independent.

The documentation clearly is claiming that the inversion is done on the signal, not on the edgeness of the dead-time (whether applied to positive or negative). So if you are claiming the latter, then the documentation is not just misleading, it is outright wrong.

@moefear85
Thanks for telling me you're misled by the documentation. We will improve it ASAP. FYI, I can add more hints in this diagram: https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/peripherals/mcpwm.html#active-high-complementary to highlight the where the inversion happens.

Okay, this is the hardware limitation:
you can't set rising edge delay for both MCPWM generator 0 and 1 at the same time.
Based on your snippet code, if we change the deadtime_config_H to
mcpwm_dead_time_config_t deadtime_config_H = {.negedge_delay_ticks = 1};

Then you can see the inverter works only for generator1, not for generator0.

We will update the programming guide to mention this limitation.

@suda-morris

Hi @moefear85 the inverting is not applied to either generator, it's applied to the "deadtime path".

Please tell me which document this drawing is from?
Thanks


Adding the cd workflow to generate releases

Description
This is a good practice to generate releases including environment variables like the RUDDER URL/KEY

@jespino Is there some context to this change?
Asking for a review from @phoinixgrr since he has been working on the CI/CD for plugins.

@crspeller the idea is to be able to generate release through the standard CI and automatically set the environment variables for the production rudder whenever the telemetry PR is merged.

@phoinixgrr so... how do you recommend to include the RUDDER key for the AI plugin? i has been added to other plugins/projects using a CI secret. How would you apply that secret here?

I would highly recommend against hardcoding the key directly into the code. Instead, consider injecting the key at runtime, although I am not sure if that's feasible in this situation. We need to understand the specifics.
Configuration settings, including keys, should be stored in the environment(injected during runtime) rather than within the code. You can refer to The Twelve-Factor App methodology for more details on best practices for configuration management.
Moreover security wise, if the Key is burned into runtime, the key can be extracted with a tool like Ghidra or strings linux command.
Let's discuss this further, to find the best way to approach this problem.
cc: @saturninoabril

Following a different path for this


Java 7 support

I'm trying to use this to build a connector between a Java 7 application and the rest of my tools communicating using nats. However, building my code on java 7 throws unsupported major/minor version 52. How difficult would it be to support use in Java 7?

@schicks , thank you for raising this issue.  We'll look into it.

Our plan is to stay on 8+, hopefully that is ok (since this bug got left behind).


ARTEMIS-200 Message Compression Support

ok. I'll close it and work on it tomorrow.
The test is MessageCompressionTest. The compression/decompression all happens at client side.

@gaohoward  you didn't need to close the PR, you could just have rebased it.. and we would merge it when it's done.

ah sorry clebert. I'll reopen this. What I'm planning to add is:

when deliver the message to non-openwire clients, the broker will decompose it as if it's a normal message.
writing some inter-protocol message tests as you suggested.

Please don't merge until I finish. Thanks

I don't think you can do that.. the message is usually converted to regular format as the message is received on the server. You will have to uncompress the message during conversion.
I suspect this would fail with these changes.

You are right clebert, I found it'd better be decompressed into regular messages before going into core, otherwise the core will have to change a lot to handle it properly.
So far I have openwire-core test almost pass (one type of messages is failing due to some bug, I'm looking into it now). Then I'll write openwire-amqp, openwire-stomp and openwire-mqtt tests.

You just need to test it once.. between openwire<->core...   the others should be right by transitivity... you just need to test between openwire compressed and core

ok got it. Thanks.

Hi clebert, it's strange the protonTest failed in this build. I don't think my changes affect that. My local build is all fine. Anyway can you take a look of my new code changes?
Thanks

I will do it after the next release cut.. on which case you will need to rebase your PR

ok.

@gaohoward  can you rebase your branch after the release?
I have run the fast tests on your branch and it was ok.. it's probably an intermittent failure on ProtonTest non related to your commit...
But I also need you to rebase since all the changes.. that will kick another test.. I will merge this tomorrow...
Please.. let me merge this.. I want to take a look at something.

ok, thanks clebert.

I'm running a few tests with your commit, if everything is ok, I will merge it tomorrow morning with a rebase.

ok


GIT commands are slow

VSCode Version:  1.7.2
OS Version: windows 7 64bit

Steps to Reproduce:
All the git operations are slow. For example, it costs about 3 seconds to stage a file in a very small repository.
I manual run the same git command:  git add -A -- testfile.txt in the terminal, it's very fast.
I run the profile in the develop tools, but the total times are small, execpt (idle) and (program).

Interesting!
Which version of git do you have it installed? Where is git.exe located?

I use git version 2.7.1.windows.2 and VSCode UI git operation is comparing to commands I usually use in the terminal, but I believe it is due to multiple git commands running by VSCode for one commit, I'm not sure but maybe this could be improved too. Because it really seems slow and not very pleasant experience.

Thanks for the feedback. We won't do much for now, but keep it in mind for the future.


Junk processing

FTL-Extract writes Junk object in every .ftl file
TODO: rework import_from_ftl to fill leave_as_is with objects, that have path. Then sort leave_as_is by path and write to correct .ftl file as is.

Resolved


Made changes to request a code review

In this milestone I:

Created a website to add and remove tasks from the to-do list.

Hello @mariegrace31🌻   ,
🎉🎉 I have checked your app and you did a great job! 🎉🎉

Highlights 🔔

You followed the JavaScript best practices correctly! 💯
You used node modules and the .lock files in the repo. 👍
You have module folder and inside it , you have various JS files. ⭐
Your project is DRY , KISS and YAGNI. 👏
Good use of type checks and naming 💯

Thanks @meryemsanem for your code review 💯


Add FreeBSD builds

Add FreeBSD builds to the release process (free with Go's cross compilation).

Thanks for the PR! I just released it.
https://github.com/seachicken/gh-poi/releases/tag/v0.5.1


fix: Recognize that the input might be a string. Fixes #2

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run; then  say 'jenkins, test it'.

jenkins, test it

Can confirm that this PR fixes #2
lgtm


Changed 'many' to 'other'.

Fixed warning 'Formatting message for language 'ar' failed with error: Message pattern is invalid.. '

Thanks!


Convert to python package

@eduardogr

@eduardogr I think it should be good to go now
I was able to get it to work on Ubuntu 18.04 with Python 3.7

It also already works on Morpheus (Ubuntu 20.04, Python 3.8,3.9, 3.10)


Hardening the authentication

Password comparison should not be done with the == operator, but ===, due to type juggling.
References

http://phpsadness.com/sad/47
turbochaos.blogspot.fr/2013/08/exploiting-exotic-bugs-php-type-juggling.html

Test case

Create an administrator with the password "240610708".
Try to login to the dashboard with the password "QNKCDZO" :-)

:+1:

Thanks for the patch!


Design pvlib.io

I'm assembling code to create pvlib.io.
What functionality is desired?  Specifically, from which formats do we want to read, and to which formats do we want to write?
I'm going to focus first on reading a few data formats. I plan to incorporate the current pvlib.tmy which reads tmy2 and tmy3 files, returning a tuple (data, metadata) where data is a pandas DataFrame and metadata is a dict.  Any comments on this design?  Personally I like it.
I have code to read surfrad files so I'll include that.

I often put metadata in a pandas Series rather than dict because this
gives me order.
On 2016-11-12 18:24, Cliff Hansen wrote:

I'm assembling code to create pvlib.io.
What functionality is desired? Specifically, from which formats do we
want to read, and to which formats do we want to write?
I'm going to focus first on reading a few data formats. I plan to
incorporate the current pvlib.tmy which reads tmy2 and tmy3 files,
returning a tuple (data, metadata) where data is a pandas DataFrame
and metadata is a dict. Any comments on this design? Personally I like it.
I have code to read surfrad files so I'll include that.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/pvlib/pvlib-python/issues/261, or mute the thread
https://github.com/notifications/unsubscribe-auth/AIlYQwSkZEiRMTP6kGe8fd9GMxsc9hXMks5q9fZFgaJpZM4KweNy.

--
Photovoltaic Performance Labs Germany
Emmy-Noether-Str. 2
79110 Freiburg
Germany
+49-761-8973-5603 (Europe)
+49-174-532-7677 (Mobile)
+1-613-817-1652 (North America)
www.pvperformancelabs.com

As @bmu once mentioned, naming our new module io is probably asking for trouble because it conflicts with the builtin python io module. I'm open to dataio, datareader, inputoutput, or other suggestions.
So far, we've only discussed creating a new module. @mikofski already complained in #235 that our modules are too long, and adding a bunch of new code to a unified io module might make it worse. I wonder if we might be better off in the long run if we instead create a subpackage comprised of one module per data type and pull the functions into the package-level namespace. This is similar to the way that the pandas io package is structured. So, we'd have something like
pvlib/
  dataio/
    __init__.py
    surfrad.py
    tmy.py
    ...

### pvlib/dataio/__init__.py ###
from pvlib.dataio.surfrad import read_surfrad
from pvlib.dataio.tmy import read_tmy2, read_tmy3

### usage ###
import pvlib
data, metadata = pvlib.dataio.read_surfrad
data2, metadata2 = pvlib.dataio.read_tmy2

This subpackage structure might make for an easy to use API and a set of easy to read/maintain modules.
Probably a good idea to stick with the (data, metadata) pattern for new reader functions.
I cannot think of a downside to returning metadata as a Series, but my instinct is to leave it as a dict (possibly an OrderedDict) and let users convert it to a Series if they want to. I seem to remember that the pandas developers recommend against using Series as a dictionary, though I can't find the reference and could be making that up. All that being said, I'm not opposed to the change.

It’s a plan, although I prefer iotools to dataio.
Function pattern to be the same as pandas, i.e., read_(format) and to_(format) as needed.
From: Will Holmgren<EMAIL_ADDRESS>Sent: Monday, November 14, 2016 12:11 PM
To: pvlib/pvlib-python<EMAIL_ADDRESS>Cc: Hansen, Clifford W<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: [EXTERNAL] Re: [pvlib/pvlib-python] Design pvlib.io (#261)
As @bmuhttps://github.com/bmu once mentioned, naming our new module io is probably asking for trouble because it conflicts with the builtin python io module. I'm open to dataio, datareader, inputoutput, or other suggestions.
So far, we've only discussed creating a new module. @mikofskihttps://github.com/mikofski already complained in #235https://github.com/pvlib/pvlib-python/issues/235 that our modules are too long, and adding a bunch of new code to a unified io module might make it worse. I wonder if we might be better off in the long run if we instead create a subpackage comprised of one module per data type and pull the functions into the package-level namespace. This is similar to the way that the pandas io package is structured. So, we'd have something like
pvlib/
dataio/
__init__.py

surfrad.py

tmy.py

...

pvlib/dataio/init.py
from pvlib.dataio.surfrad import read_surfrad
from pvlib.dataio.tmy import read_tmy2, read_tmy3
usage
import pvlib
data, metadata = pvlib.dataio.read_surfrad
data2, metadata2 = pvlib.dataio.read_tmy2
This subpackage structure might make for an easy to use API and a set of easy to read/maintain modules.
Probably a good idea to stick with the (data, metadata) pattern for new reader functions.
I cannot think of a downside to returning metadata as a Series, but my instinct is to leave it as a dict (possibly an OrderedDict) and let users convert it to a Series if they want to. I seem to remember that the pandas developers recommend against using Series as a dictionary, though I can't find the reference and could be making that up. All that being said, I'm not opposed to the change.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/pvlib/pvlib-python/issues/261#issuecomment-260430954, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFJNL62IuuG0CilpHmiCpwMd105VXLI0ks5q-LJZgaJpZM4KweNy.

I defer to people with more experience on the code organization, but I do suspect there will be a few potentially shared elements.  What I could imagine--and would really like to have--is a kind of enhanced read_csv that can read extra information in the header and/or footer in addition to the regular columnar data. Many file formats could build on that just by setting parameters and translating column or key names.  An option could be to return the header and/or footer as a text block to be parsed later.

@adriesse do you have an example file & code?
How would you implement with the file irradiation-0e2a19f2-abe7-11e5-a880-5254002dbd9b.csv
In most cases, the input will requrie an adaption of the dateparser to the specific data format. This usually gets you going.
Or would the scope in addition include quality checks or reformatting on the data?
I am still not clear what the output is desired to be.

A dataframe as read in?
A trying to comply closest with Variables and Symbols?

In the latter case, what do we with additional variables e.g. in spectral datasets or in BSRN files, for example?

Hi all, sorry I haven't chimed in. I already have tmy2 and tmy3 readers. I'll send them when I get in.
AFA the dataio module and namespace question.

I also like dataio.
Will, I like your idea of sub packages, I think it's the easiest way to decrease the omnipotence of the god modules. Just change irradiance.py to a folder called irradiance and put an __init__.py in it. Then import the symbols that were originally imported directly from irradiance.py but are now in modules like irradiance/spa.py
Anton, clashes in names should be rare in Python because it uses namespaces. EG pvlib.data.io can't be confused with the built-in io because the long name that precedes it.

Mark, I am confused about your tmy readers. Do they do the same thing as pvlib's existing tmy readers?
Let's take things one step at a time and have Cliff just make us a dataio/iotools subpackage for the IO capabilities. I have been thinking about larger reorganizations, though, will eventually comment more in #235.
If necessary, the subpackage could have a core.py or its own tools.py module for code that is reused throughout the package.
I want to keep the bar for contributing to pvlib as low as possible, so it's fine with me if different modules do different amounts of processing to their respective data formats. Some might be 1000 line monsters that do QA and return data that is in "pvlib conventions," while others might not do much more than call pd.read_csv with a couple of arguments that are unique to that data format.

RE: TMY
@wholmgren , sorry I misunderstood the intention of the proposed dataio, and also I wasn't aware of the readtmy2 and readtmy3 methods in tmy.py. So please disregard (2) in my comment above.
RE: reorganization
I agree with you that the bar for contributions should be low. Your ideas for core.py and tools.py sound good, although if I understand correctly (?) that approach may break some code if it shift things from say pvlib.irradiance to pvlib.core.irradiance so that's something to consider. IMHO changing the top level modules to subpackages would preserve the current API, but allow more flexibility. See my comment on #235
RE: dataio
Cliff, some ideas for formats that could be read are:

SAM module and inverter Excel spreadsheets - I think this is already covered by pvsystem.retrieve_sam method, so similar to TMY methods, would you relocate them to dataio?
NSRDB API for weather data
If I can manage to spend a little time to upload the rest of the module and inverter data to pvfree  it would be nice to supplement retrieve_sam.
PAN and ONB file reader, although I don't know if there are any methods in pvlib currently that would utilize these, I know a lot of people would appreciate being able to load these into Python for use with custom tools.
Other data sources of irradiance, atmospheric and weather measurements would be very useful. These come from numerous sources, so maybe they could be contributed by users as needed. Here's a short list of some examples:

AERONET
MERRA
ECMWF
Solar Prospector
Clean Power Research
3-Tier Vaisala
Solar GIS
Meteonorm/Meteotest
SoDa-Pro

@cwhanse / @mikofski many items of the last comments are already in https://github.com/pvlib/pvlib-python/issues/29 are they complementary? shall the other be closed?
I wonder if there are any observations regarding the comments in https://github.com/pvlib/pvlib-python/issues/261#issuecomment-260628285
Otherwise, who will provide a prototype or will have a PR each one with the data set reader they have implemented?

@cwhanse please have a look at the PR
The data is here:
cd MY-GITHUB-CLONES
git clone https://github.com/dacoex/pvlib_data

@cwhanse & @adriesse my refractored PR is up:

maccrad https://github.com/pvlib/pvlib-python/pull/279
pvsyst https://github.com/pvlib/pvlib-python/pull/280

@wholmgren seems to lead towards keeping the iotools simple and as a customised wrapper for pandas.read_csv
Shall iotools/util also provide functions to:

read metadata, usually in the lines before column header: coordinates, time reference?
localise the UTC based timeseries as suggested: 4 days ago

So what shall be the outputs?

a dataframe with the raw data?
a dataframe with renamed columns to match pvlib convention?
a metadata dictionary?
a location with timezone, which involves usually retrieving this info either from geonames or using an additional package.
a dataframe with renamed columns to match pvlib convention & localised index?

I personally prefer to have the library do as much as possible:

read data
reformat data
prepare a Location
localise data

I am looking forward to your feedback & will then modify my PRs according to what seems to be consensus.

I understand why you would like to handle the metadata with this library.
I wonder if the problem is that not all file types include all metadata?
For example the PVsyst output file doesn't include any timezone or lat/long
information, because it just references a .SIT file that includes that
information. Not sure how to handle this other than to have the user define
a Location separately to use.
Jessica
On Mon, Dec 5, 2016 at 8:57 AM, DaCoEx<EMAIL_ADDRESS>wrote:

@cwhanse https://github.com/cwhanse & @adriesse
https://github.com/adriesse my refractored PR is up:

maccrad #279 https://github.com/pvlib/pvlib-python/pull/279
pvsyst #280 https://github.com/pvlib/pvlib-python/pull/280

@wholmgren https://github.com/wholmgren seems to lead towards keeping
the iotools simple and as a customised wrapper for pandas.read_csv
Shall iotools/util also provide functions to:

read metadata, usually in the lines before column header:
coordinates, time reference?
localise the UTC based timeseries as suggested: 4 days ago
https://github.com/pvlib/pvlib-python/pull/270#issuecomment-264102908

So what shall be the outputs?

a dataframe with the raw data?
a dataframe with renamed columns to match pvlib convention?
a metadata dictionary?
a location with timezone, which involves usually retrieving this
info either from geonames or using an additional package.
a dataframe with renamed columns to match pvlib convention &
localised index?

I personally prefer to have the library do as much as possible:

read data
reformat data
prepare a Location
localise data

I am looking forward to your feedback & will then modify my PRs according
to what seems to be consensus.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/pvlib/pvlib-python/issues/261#issuecomment-264910082,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AH66AcfaCRnSZDxoPpBb9fNqf6VM-NVaks5rFEKIgaJpZM4KweNy
.

I wonder if the problem is that not all file types include all metadata?

Yes.
But most scientific providers at least include information on time reference, i.e. UTC.
So with this and the coordinates, we could derive the timezine and calculate the location for the typical input meto files.
Timezone is derived either by web query to geonames or local libraries.

For example the PVsyst output file doesn't include any timezone or lat/long information, because it just >references a .SIT file that includes that information. Not sure how to handle this other than to have the >user define a Location separately to use.

Correct. Actually, in the case of PVSyst, I would assume that a location exists because PVSyst hourly output usually does not include GHI. I would assume that a pvlib user uses this data to compare the result of both modelling environments.
So my current proposal would be address in iotools:

Standard output of a pvlib.iotools.FORMAT reader would be a tuple with

dataframe with raw data (for comparison & debugging)
dataframe with renamed columns to match pvlib convention
metadata dictionary as suggested by @wholmgren in iotools: reader for maccrad #279

pvlib.iotools.util to include some tools

optional tool to retrieve the timezone
optional tool to localise dataframe
further functions, e.g. checker if the radiation starts before sunrise to inform that there is a timezone issue

So depening on the dataset, the user coud employ the functions in util for more convenience.
The capabilities of each format / reader could be documented in the docstring.

Yes, having tools available for the user to apply sounds like the right
approach.
On Tue, Dec 6, 2016 at 2:05 AM, DaCoEx<EMAIL_ADDRESS>wrote:

I wonder if the problem is that not all file types include all metadata?
Yes.
But most scientific providers at least include information on time
reference, i.e. UTC.
So with this and the coordinates, we could derive the timezine and
calculate the location for the typical input meto files.
Timezone is derived either by web query to geonames or local libraries.
For example the PVsyst output file doesn't include any timezone or
lat/long information, because it just >references a .SIT file that includes
that information. Not sure how to handle this other than to have the >user
define a Location separately to use.
Correct. Actually, in the case of PVSyst, I would assume that a location
exists because PVSyst hourly output usually does not include GHI. I would
assume that a pvlib user uses this data to compare the result of both
modelling environments.
So my current proposal would be address in iotools:

Standard output of a pvlib.iotools.FORMAT reader would be a tuple
with

dataframe with raw data (for comparison & debugging)
dataframe with renamed columns to match pvlib convention
metadata dictionary as suggested by @wholmgren
https://github.com/wholmgren in iotools: reader for maccrad #279
https://github.com/pvlib/pvlib-python/pull/279#discussion_r90881672

pvlib.iotools.util to include some tools

optional tool to retrieve the timezone
optional tool to localise dataframe
further functions, e.g. checker if the radiation starts before
sunrise to inform that there is a timezone issue

So depening on the dataset, the user coud employ the functions in util
for more convenience.
The capabilities of each format / reader could be documented in the
docstring.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/pvlib/pvlib-python/issues/261#issuecomment-265110051,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AH66AdRkKTYFZ3SIqrAi3gUMElcw5CQXks5rFTOAgaJpZM4KweNy
.

try tzwhere to determine timezone from coordinates:
from tzwhere import tzwhere
from datetime import datetime
import pytz

# timezone lookup, force nearest tz for coords outside of polygons
WHERETZ = tzwhere.tzwhere(shapely=True, forceTZ=True)
# daylight savings time (DST) in northern hemisphere starts in March and ends
# in November and the opposite in southern hemisphere
JAN1 = datetime(2016, 1, 1)  # date with standard time in northern hemisphere
JUN1 = datetime(2016, 6, 1)  # date with standard time in southern hemisphere

# notes and links on Python datetime tzinfo dst:
# http://stackoverflow.com/questions/17173298/is-a-specific-timezone-using-dst-right-now
# http://stackoverflow.com/questions/19774709/use-python-to-find-out-if-a-timezone-currently-in-daylight-savings-time

# methods for determining DST vs. STD
# tz.localize(JAN1).timetuple().tm_isdst -> boolean, True if DST, False if STD
# tz.localize(JAN1).dst() -> timedelta, 0 if STD
# tz.dst(JAN1) -> timedelta, 0 if STD

def tz_latlon(lat, lon):
    """
    Timezone from latitude and longitude.

    :param lat: latitude [deg]
    :type lat: float
    :param lon: longitude [deg]
    :type lon: float
    :return: timezone
    :rtype: float
    """

    # get name of time zone using tzwhere, force to nearest tz
    tz_name = WHERETZ.tzNameAt(lat, lon, forceTZ=True)

    # check if coordinates are over international waters
    if not tz_name or tz_name in ('uninhabited', 'unknown'):
        # coordinates over international waters only depend on longitude
        return lon // 15.0
    else:
        tz_info = pytz.timezone(tz_name)  # get tzinfo

    # get the daylight savings time timedelta
    tz_date = JAN1  # standard time in northern hemisphere
    if tz_info.dst(tz_date):
        # if DST timedelta is not zero, then it must be southern hemisphere
        tz_date = JUN1  # a standard time in southern hemisphere
    tz_str = tz_info.localize(tz_date).strftime('%z')  # output timezone from ISO
    # convert ISO timezone string to float, including partial timezones
    return float(tz_str[:3]) + float(tz_str[3:]) / 60.0, tz_info

if __name__ == "__main__":
    # test tz_latlon at San Francisco (GMT+8.0)
    gmt, tz_info = tz_latlon(37.7, -122.4)
    assert gmt == -8.0
    assert tz_info.zone == 'America/Los_Angeles'
    assert tz_info.utcoffset(JAN1).total_seconds()/3600.0 == -8.0
    assert tz_info.utcoffset(JUN1).total_seconds()/3600.0 == -7.0
    # New_Delhi, India (GMT-5.5)
    gmt, tz_info = tz_latlon(28.6, 77.1)
    assert gmt == 5.5
    assert tz_info.zone == 'Asia/Kolkata'
    assert tz_info.utcoffset(JAN1).total_seconds()/3600.0 == 5.5
    assert tz_info.utcoffset(JUN1).total_seconds()/3600.0 == 5.5
    # also works with integers (EG: Santa Cruz, CA)
    gmt, tz_info = tz_latlon(37, -122)
    assert gmt == -8.0
    assert tz_info.zone == 'America/Los_Angeles'

You can call this like in the script:
gmt, tz_info = tz_latlon(37, -122)
# gmt: -8.0
# tz_info: <DstTzInfo 'America/Los_Angeles' LMT-1 day, 16:07:00 STD>

Also see:
Difference between timezones America/Los_Angeles and US/Pacific and PST8PDT?

Warning
Please do not use any of the timezones listed here in "other timezones" (besides UTC), they only exist for backward compatible reasons, and may expose erroneous behavior.

to the reverse geopy with ESRI ArcGIS works great. Registration with ESRI developer or ArcGIS public account is free.

@jforbess so I conclude you agree to the structure described above in issuecomment-265110051?
@mikofski Thanks for the code.
I used a simpler library: iotools: reader for maccrad by dacoex · Pull Request #279 · pvlib/pvlib-python
But @wholmgren did not like the addition of an external dependency:
issuecomment-264486119
This is why I propose above to make it optional.
Well, if there are no further ideas or suggestions, I will revise the PR according to the discussion.

@dacoex, yes, and I see that @wholmgren didn't like an iotools/util
function to localize tz, instead recommending using the pandas function,
which I understand, but part of me thinks that having a function in the api
will help users be consistent in their usage. This may be a philosophical
argument?
On Tue, Dec 6, 2016 at 2:35 PM, DaCoEx<EMAIL_ADDRESS>wrote:

@jforbess https://github.com/jforbess so I conclude you agree to the
structure described above in issuecomment-265110051
https://github.com/pvlib/pvlib-python/issues/261#issuecomment-265110051?
@mikofski https://github.com/mikofski Thanks for the code.
I used a simpler library: iotools: reader for maccrad by dacoex · Pull
Request #279 · pvlib/pvlib-python
https://github.com/pvlib/pvlib-python/pull/279/files/112b7bddc8e9ab554d01f8d79178d56d975e11b4#diff-88fd35c9724ff076feeb9cae8cebeeb9R10
But @wholmgren https://github.com/wholmgren did not like the addition
of an external dependency:
issuecomment-264486119
https://github.com/pvlib/pvlib-python/pull/274#issuecomment-264486119
This is why I propose above to make it optional.
Well, if there are no further ideas or suggestions, I will revise the PR
according to the discussion.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pvlib/pvlib-python/issues/261#issuecomment-265295207,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AH66AWrm97XkPsAeZ6GTW1woo3H2KgDIks5rFeMrgaJpZM4KweNy
.

@dacoex I like timezonefinder please use pypi reference not github.
I disagree with @wholmgren #274, IMO it's fine to have dependencies, as long as they are mature, well documented and widely used, which timezonefinder is, since it's on pypi, v1.5.7 has over 2000 downloads, it's based on tzwhere, the previous best choice, it's been recently updated and had a steady record of releases, etc.
IMHO using open source is one of the reasons to use Python. PyPI is one of the reasons Python is so powerful. Adding dependencies instead of rolling your own makes your code stronger.
Perhaps you can selectively let users use (or not) the import by putting it in a try: except: block.
Although, if possible perhaps it's good to try to make the dependency arbitrary, so you could switch it later. For example
def get _latlon(lat, lon, method, **kw):
    """Implemented timezone finder"""
    if method is None:
        try:
            import timezonefinder
        except:
            return lon // 15.0
        else:
            return timezonefinder(lat, lon, **kw)
    else:
        return method(lat, lon, **kw)

Thanks to all for their feedback. I will propose an improved version and also add solargis format.
Just give me some time because I just got loaded with a bit of work.

Here is another thought:
There are actually two kinds of meta data in your maccrad file: per-file metadata and per-column meta data (in this case descriptions and units).  I have seen this in some other file formats as well, and it could be something worth reading and returning separately.

So what shall be the outputs?
a dataframe with the raw data?

I don't know what you mean by raw data in this context. The columns are unchanged?

a dataframe with renamed columns to match pvlib convention?

Probably, though would be great if this was an option with a rename=True default. Not sure if we should also change the existing tmy readers to be more consistent.

a metadata dictionary?

readtmy2 and readtmy3 return a tuple of metadata, data. Seems reasonable to me. Has anyone been unhappy with this in the past or wished that those readers did more?

a location...

I strongly oppose this. I think it is essential that pvlib's class layer lives above pvlib's functional layer, though, I will take all the blame if the distinction or the reasons for it are unclear. Returning a Location object would ruin the distinction. As discussed elsewhere, you could instead add a Location.from_maccrad(metadata) method that does the job while retaining a clean separation among modules and between the class/functional layers. See Location.from_tmy for inspiration. Location.from_maccrad(file) could even be possible.

a dataframe with renamed columns to match pvlib convention & localised index?

I think that an IO reader should return a dataframe that is localized to the format of the timestamps of the file and/or the timezone specific metadata of the file. That is: if the timestamps say MST or -0700 then the dataframe should be localized accordingly, or, if the timestamps are ambiguous but the metadata says e.g. tz=MST then the dataframe should be localized. pvlib should not guess at the localization of the data.
Fine with me if you want to add a Location.guess_tz or similar method to Location. Within that method, you can import and run the optional library of your choosing or make a request to the google api.

pvlib.iotools.util to include some tools
optional tool to retrieve the timezone

I can see an argument for Location.guess_tz() with no arguments since it would look up its own self.latitude and self.longitude. Otherwise, I think it's better for users to make their own call to a library that gets the tz with one simple line of code.

optional tool to localise dataframe

I'm confused about the arguments surrounding creating our own localize/convert functions. pandas has a tz_localize method and a tz_convert method. They do exactly what their names suggest. I don't know how we can improve upon that.

further functions, e.g. checker if the radiation starts before sunrise to inform that there is a timezone issue

Sounds complicated to do reliably and probably unnecessary.

See Location.from_tmy for inspiration. Location.from_maccrad(file) could even be possible.

OK.

I don't know what you mean by raw data in this context. The columns are unchanged?

Yes, just the result of read_csv.

I can see an argument for Location.guess_tz() with no arguments since it would look up its own >self.latitude and self.longitude.

Where would that be placed?
If put in location.py it would add more overhead to this module. If kept in iotools/util, this could be an optional module, i.e. not loaded on default by iotools/api.
I still prefer to include a few shortcut functions in iotools/util.
At least those which make sense.

Otherwise, I think it's better for users to make their own call to a library >that gets the tz with one >simple line of code.

Even the reader can be done by the user. But isn't this here also to make life more simple?
If we are guessing the tz for most of the files, this it could be generalised in a common function to reduce the number of copy&paste code.
I would prefer to let the software do whatever can be done automatically. The result is to be verified anyway.

pandas has a tz_localize method and a tz_convert method. They do exactly what their names >suggest. I don't know how we can improve upon that.

Well, I took mine from pandas doc. but maybe they'd changed that api again?
@adriesse  I suggest to add the column metadata to the docstring as in
https://github.com/pvlib/pvlib-python/blob/io/pvlib/iotools/tmy.py#L73
This is also because it is not used for further calculations
Summarising, the next revision fill be structured more closely like the existing tmy readers.
This was totally overlooked in my initial code.
Sorry if that had spurred an unnecessary discussion. But seems that it helped to get a common about this module.
As a learning, we may add a specification and some instructions on how to add a new reader to the docs.

On Wed, Dec 7, 2016 at 8:31 AM, Will Holmgren<EMAIL_ADDRESS>wrote

further functions, e.g. checker if the radiation starts before sunrise to
inform that there is a timezone issue
Sounds complicated to do reliably and probably unnecessary.

This may be unnecessary for standard files, but I have been wanting it for
all of the data that I get from SCADA systems. Just found a system that
didn't apply Daylight Savings on the actual day, but a week early. But only
in the spring. And only the first two years. Otherwise, it had the right
alignment with Daylight Savings.
I spent a lot of time wrapping my head around the right way to handle
timezones because of daylight savings and the fact that my data sometimes
comes with timestamps from a timezone that it is not located in. (A client
pulls data in US/Eastern for a plant that is in California. The SCADA
thinks it is doing the right thing, maybe, because it is relative to where
the data is being queried, but it is not the right thing at all.)
But I admit, this shouldn't be an issue for any standard file that gets a
standard reader. But it is critical if anyone tries to generalize iotools
for a somewhat standard csv.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pvlib/pvlib-python/issues/261#issuecomment-265496432,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AH66ATjzki4jLt5hI3u28wrM7Nxkl5plks5rFt9ngaJpZM4KweNy
.

yes, @jforbess is right. this also applies to most datalogger files...

This issue seems a bit outdated, but anyway, I have just written a short function to read epw weather files from EnergyPlus. Note that the output is not harmonized with the output of the TMY functions.
Feel free to re-use or integrate into the library:
`
def readepw(filename=None):
    '''
    Reads an EPW file into a pandas dataframe.

    Function tested with EnergyPlus weather data files:
    https://energyplus.net/weather

    Parameters
    ----------
    filename : None or string
        If None, attempts to use a Tkinter file browser. A string can be
        a relative file path, absolute file path, or url.

    Returns
    -------
    Tuple of the form (data, metadata).

    data : DataFrame
        A pandas dataframe with the columns described in the table
        below.

    metadata : dict
        The site metadata available in the file.

    Notes
    -----

    The returned structures have the following fields.

    =======================================================================
    Data field
    =======================================================================
    Datetime data
    Dry bulb temperature in Celsius at indicated time
    Dew point temperature in Celsius at indicated time
    Relative humidity in percent at indicated time
    Atmospheric station pressure in Pa at indicated time
    Extraterrestrial horizontal radiation in Wh/m2
    Extraterrestrial direct normal radiation in Wh/m2
    Horizontal infrared radiation intensity in Wh/m2
    Global horizontal radiation in Wh/m2
    Direct normal radiation in Wh/m2
    Diffuse horizontal radiation in Wh/m2
    Averaged global horizontal illuminance in lux during minutes preceding the indicated time
    Direct normal illuminance in lux during minutes preceding the indicated time
    Diffuse horizontal illuminance in lux  during minutes preceding the indicated time
    Zenith luminance in Cd/m2 during minutes preceding the indicated time
    Wind direction at indicated time. N=0, E=90, S=180, W=270
    Wind speed in m/s at indicated time
    Total sky cover at indicated time
    Opaque sky cover at indicated time
    Visibility in km at indicated time
    Ceiling height in m
    Present weather observation
    Present weather codes
    Precipitable water in mm
    Aerosol optical depth
    Snow depth in cm
    Days since last snowfall
    Albedo
    Liquid precipitation depth in mm at indicated time
    Liquid precipitation quantity
    =======================================================================

    ===============   ======  ===================
    key               format  description
    ===============   ======  ===================
    altitude          Float   site elevation
    latitude          Float   site latitudeitude
    longitude         Float   site longitudeitude
    Name              String  site name
    State             String  state
    TZ                Float   UTC offset
    USAF              Int     USAF identifier
    ===============   ======  ===================

    S. Quoilin, October 2017
    '''

    def _interactive_load():
        import Tkinter
        from tkFileDialog import askopenfilename
        Tkinter.Tk().withdraw() #Start interactive file input
        return askopenfilename()

    if filename is None:
        try:
            filename = _interactive_load()
        except:
            raise Exception('Interactive load failed. Tkinter not supported on this system. Try installing X-Quartz and reloading')

    head = ['dummy0', 'Name', 'dummy1', 'State', 'dummy2', 'USAF', 'latitude', 'longitude', 'TZ', 'altitude']

    csvdata = open(filename, 'r')

    # read in file metadata
    temp = dict(zip(head, csvdata.readline().rstrip('\n').split(",")))

    # convert metadata strings to numeric types
    meta = {}
    meta['Name'] = temp['Name']
    meta['State'] = temp['State']
    meta['altitude'] = float(temp['altitude'])
    meta['latitude'] = float(temp['latitude'])
    meta['longitude'] = float(temp['longitude'])
    meta['TZ'] = float(temp['TZ'])
    meta['USAF'] = int(temp['USAF'])

    headers = ["year","month","day","hour","min","Dry bulb temperature in C","Dew point temperature in C","Relative humidity in percent","Atmospheric pressure in Pa","Extraterrestrial horizontal radiation in Wh/m2","Extraterrestrial direct normal radiation in Wh/m2","Horizontal infrared radiation intensity in Wh/m2","Global horizontal radiation in Wh/m2","Direct normal radiation in Wh/m2","Diffuse horizontal radiation in Wh/m2","Averaged global horizontal illuminance in lux during minutes preceding the indicated time","Direct normal illuminance in lux during minutes preceding the indicated time","Diffuse horizontal illuminance in lux  during minutes preceding the indicated time","Zenith luminance in Cd/m2 during minutes preceding the indicated time","Wind direction. N=0, E=90, S=180, W=270","Wind speed in m/s","Total sky cover","Opaque sky cover","Visibility in km","Ceiling height in m","Present weather observation","Present weather codes","Precipitable water in mm","Aerosol optical depth","Snow depth in cm","Days since last snowfall","Albedo","Liquid precipitation depth in mm","Liquid precipitation quantity"]
    Data = pd.read_csv(filename, skiprows=8,header=None)
    del Data[5]
    Data.columns = headers
    Data.index = pd.to_datetime(Data[["year","month","day","hour"]])

    Data = Data.tz_localize(int(meta['TZ']*3600))

    return Data, meta

@squoilin  - Silvain, this EPW reader function is quite useful.  I'm incorporating it into my development of bifacialvf and bifacial_radiance at github.com/nrel/bifacialvf and github.com/nrel/bifacial_radiance.   If/ when this gets pulled into the pvlib distribution, I'll switch over to the 'official' version.
Thanks!


Q-Net implemented using Keras does not to learn, and TF does

Hey guys, I observed an intriguing fact when implementing a [Q-Network] (https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf) using Keras. Hope you can help me to figure out what is going on.
Basically, when I train the model using Keras, it does not learn anything, but when I train the exactly same model using Tensorflow it works fine. The figure below shows it. In the figure, the x-axis is the number of episodes; the red line is 1 when the agent reached the goal in the episode, and 0 otherwise; the blue line is the percentage of times the agent reached the goal in the last 30 episoes (higher is better); and the green line represents how many random actions the agent have taken (1 = 100% => all actions are random, and 0 = 0% => all actions are greedy).

The differences between the codes are just the model definition, and the methods to forward, and to train the networks. I have no idea why the results are not equivalent. If someone can give me some feedback on the following investigative questions, I would be really grateful.

Are there any problem in using a Keras model inside a class?
Are predict_on_batch, and train_on_batch restricted to classification problems or they can be used for regression? Reinforcement learning using Q-nets is a regression problem.
Are the executions of predict_on_batch, and train_on_batch independent of each other? Is it OK to train some data from the training set using train_on_batch, and test data from the testing set using predict_on_batch iteratively? Or information (gradients, model variables, etc.) are shared between calls of these methods?

The code is quite small and simple. It is available at:
[link to code] (https://gist.github.com/filipemtz/3176165e9b22e2f8d992ce9f1fd0adfe)
It contains 3 files:

main.py : it loads the environment and, iteratively, executes an episode, and train the network.
qnet.py: class containing the model. You can choose to use Keras ou Tensorflow by setting the flag USE_KERAS to True or False.
slow_avg.py: utility code to compute the percentage of times the agent reached the goal in the last 30 episodes.

To run the code, use:
python main.py > report.txt
To plot the graph (run from the same dir of the code):
gnuplot
[in the gnuplot prompt:]
!grep Episode report.txt > a.txt && python slow_avg.py > b.txt
plot './a.txt' u 11 w l, './a.txt' u 4 w l axis x1y2, './b.txt' u 4 w l

Take a look at keras-rl if you want to get into RL with Keras. I'm surprised either one is training. You generate the q for each input, change one element of the q, then use that to train the model. You are only seeing one thing, but you are training as if you saw everything.
Your learning rate for tf is 100x the rate for Keras, which might have something to do with the difference.
The standard approach is to have your model inputs be the observation and the action. Q is calculated based on the observation. One of the Q values is selected using the action. Then, only that Q value is trained. If you read through the keras-rl source you should get a sense of how to put it together.
Cheers

@bstriner thanks for the suggestion about keras-rl. I will check that.
About the fact that I change just one element of the q and then retrain, it works because for all output neurons but the updated one, the network prediction will be equal to the target, and so the gradients for them will be zero. It is inefficient because I need to perform a forward each time a train the net, but I was accepting that to keep the code simple.
Unfortunately, even raising Keras' learning rate the results still nearly the same.
What is more intriguing is why the results are so different if the model and the training process are the same...


Return non-zero exit code on task failure

Is there a way to get the luigi CLI to return non-zero exit codes on task failures? It seems by default it only prints the task status in human readable form to stderr. We would like our scripts to know if the tasks succeeded, but luigi returns 0 even if tasks have failed.
To work around this we have created a wrapper script as follows:
# get the luigi results from stderr
results="$(luigi "${@:1}" 2>&1)"

# print the results for debugging
echo "$results"

# grep the results to return a proper exit code
echo "$results" | grep 'This progress looks :)'

Is there a better way? Or would you be open to a PR to add this functionality?

https://luigi.readthedocs.io/en/stable/configuration.html#retcode I think this might help you out

Aha, this does seem like what I'm looking for. Thanks!

Sorry to reopen this issue, but @henryrizzi suggestion doesn't work for me.
I don't know if I miss understood the feature, but even adding the section [retcode] to my luigi.cfg, if I do:
python pypeline.py task --workers 4 echo $?
The echo always returns 0 even when the task fails. Shouldn't it return the error described in [retcode] section on luigi.cfg?

I was facing this problem even with [retcode] section in my luigi.cfg file. I have just found out the reason was using luigi.run() instead of luigi command line tool. I think return codes don't work with luigi.run() for some reason. You can try:
export PYTHONPATH=$PYTHONPATH:~
luigi pypeline task --workers 4
echo $?